Article

From:
To:
Samuel Aeschbacher
Subject:
Re: Out of memory Exception when using large StringStream.DataString inD2009
Newsgroup:
embarcadero.public.delphi.rtl

Re: Out of memory Exception when using large StringStream.DataString inD2009

> {quote:title=Samuel Aeschbacher wrote:}{quote}
> Hello 
> I have to read large text-files (> 100 MB) into memory, scan the content and update a database. 
> 
> I use TFileStream to read the data, then copying the filestream to a stringstream for better string handling. 
> If I seek a substring in Stringstream.DataString, I get an "Out-of-memory-Exception" if the size of the Stream exceeds 200 MB (it worked well with a size <= 136 MB and crashed with a size >= 280 MB). 
> 
> I'm working with D2009, Windows 7 (32bit and 64bit), Windows-Server 2008 R2 
> 
> the filestream (fs) and the stringstream (strstrm) are private class variables of the TDLHI_Main class. 
> 
> Here the code: 
> 
> function TDLHI_Main.ImportKompendium(var MsgList: TListBox; var PrgrssBar: TProgressBar; FileName : string): string; 
> var 
>   i : integer; 
> begin 
>   Result := ''; 
>   fs := TFileStream.Create(FileName, fmOpenRead + fmShareDenyWrite); 
>   strstrm := TStringStream.Create(''); 
>   strstrm.CopyFrom(fs, fs.Size); 
>   fs.Free; 
>   i := pos ('<OK_ERROR>', strstrm.DataString); *// ==> And on the this line, I get the out-of-memory-exception!* 
> 
> It's interesting, to get the exception on the "pos"-Statement and not as I expected on the copying of the stream. 
> 
> In D2007, it worked well; in D2009 I get this error on the mentioned Windows-platforms (I didn't try to run it on other windows-versions) 
> 

Your text file is converted to unicode and it holds twice as data now. since you were
using Delphi 2007 before, I suspect that your text file is not unicode.
Instead of TStringStream try this class (I haven't fully tested it but it should work)

{code}

  TRawStringStream = class(TBytesStream)   private     function GetDataString: RawByteString;   public     constructor Create; overload;     constructor Create(const AString: RawByteString); overload;     constructor Create(const ABytes: TBytes); overload;     function ReadString(Count: Longint): RawByteString;     procedure WriteString(const AString: RawByteString);     property DataString: RawByteString read GetDataString;   end;
constructor TRawStringStream.Create; begin   Create(''); end;
constructor TRawStringStream.Create(const AString: RawByteString); begin   inherited Create(BytesOf(AString)); end;
constructor TRawStringStream.Create(const ABytes: TBytes); begin   inherited Create(ABytes); end;
function TRawStringStream.GetDataString: RawByteString; begin   if Size = 0 then Result := ''   else     begin      SetLength(Result, Size);      Move(Bytes[0], Result[1], Size);     end; end;
function TRawStringStream.ReadString(Count: Integer): RawByteString; begin   if Count > Size - Position then Count := Size - Position;   if Count = 0 then Result := ''   else     begin      SetLength(Result, Count);      Move(Bytes[Position], Result[1], Count);      Position := Position + Count;     end; end;
procedure TRawStringStream.WriteString(const AString: RawByteString); var   LBytes: TBytes; begin   LBytes := BytesOf(AString);   Write(LBytes[0], Length(LBytes)); end;
{code}
Also you cannot use Pos like you used to because it will also trigger Unicode conversion. You have to declare RawByteString variable and assign it before calling Pos function.

{code}
var substr: RawByteString;

..... substr := '<OK_ERROR>'; i := pos (substr, strstrm.DataString);
{code}
Dalija Prasnikar
FYI: Phrase searches are enclosed in either single or double quotes
 
 
Originally created by
Tamarack Associates
Sun, 24 Nov 2024 13:18:38 UTC
Copyright © 2009-2024
HREF Tools Corp.