> {quote:title=Samuel Aeschbacher wrote:}{quote}
> Hello
> I have to read large text-files (> 100 MB) into memory, scan the content and update a database.
>
> I use TFileStream to read the data, then copying the filestream to a stringstream for better string handling.
> If I seek a substring in Stringstream.DataString, I get an "Out-of-memory-Exception" if the size of the Stream exceeds 200 MB (it worked well with a size <= 136 MB and crashed with a size >= 280 MB).
>
> I'm working with D2009, Windows 7 (32bit and 64bit), Windows-Server 2008 R2
>
> the filestream (fs) and the stringstream (strstrm) are private class variables of the TDLHI_Main class.
>
> Here the code:
>
> function TDLHI_Main.ImportKompendium(var MsgList: TListBox; var PrgrssBar: TProgressBar; FileName : string): string;
> var
> i : integer;
> begin
> Result := '';
> fs := TFileStream.Create(FileName, fmOpenRead + fmShareDenyWrite);
> strstrm := TStringStream.Create('');
> strstrm.CopyFrom(fs, fs.Size);
> fs.Free;
> i := pos ('<OK_ERROR>', strstrm.DataString); *// ==> And on the this line, I get the out-of-memory-exception!*
>
> It's interesting, to get the exception on the "pos"-Statement and not as I expected on the copying of the stream.
>
> In D2007, it worked well; in D2009 I get this error on the mentioned Windows-platforms (I didn't try to run it on other windows-versions)
>
Your text file is converted to unicode and it holds twice as data now. since you were
using Delphi 2007 before, I suspect that your text file is not unicode.
Instead of TStringStream try this class (I haven't fully tested it but it should work)
{code}
TRawStringStream = class(TBytesStream)
private
function GetDataString: RawByteString;
public
constructor Create; overload;
constructor Create(const AString: RawByteString); overload;
constructor Create(const ABytes: TBytes); overload;
function ReadString(Count: Longint): RawByteString;
procedure WriteString(const AString: RawByteString);
property DataString: RawByteString read GetDataString;
end;
constructor TRawStringStream.Create;
begin
Create('');
end;
constructor TRawStringStream.Create(const AString: RawByteString);
begin
inherited Create(BytesOf(AString));
end;
constructor TRawStringStream.Create(const ABytes: TBytes);
begin
inherited Create(ABytes);
end;
function TRawStringStream.GetDataString: RawByteString;
begin
if Size = 0 then Result := ''
else
begin
SetLength(Result, Size);
Move(Bytes[0], Result[1], Size);
end;
end;
function TRawStringStream.ReadString(Count: Integer): RawByteString;
begin
if Count > Size - Position then Count := Size - Position;
if Count = 0 then Result := ''
else
begin
SetLength(Result, Count);
Move(Bytes[Position], Result[1], Count);
Position := Position + Count;
end;
end;
procedure TRawStringStream.WriteString(const AString: RawByteString);
var
LBytes: TBytes;
begin
LBytes := BytesOf(AString);
Write(LBytes[0], Length(LBytes));
end;
{code}
Also you cannot use Pos like you used to because it will also trigger Unicode conversion.
You have to declare RawByteString variable and assign it before calling Pos function.
{code}
var substr: RawByteString;
.....
substr := '<OK_ERROR>';
i := pos (substr, strstrm.DataString);
{code}
Dalija Prasnikar