Split large 2 GB text files

[Stuartf]

Hi All

I have a large text file (which I receive once a month) which is over 2gig and has about 30 million names and addresses

I need to split the file say into 5 chunks by line number then I can import into my database app. ( I normally do this manually)

I know there is software out there to do this but hey If i can do it for free and learn something in the process then hey Ill at least try.

I've not used macro's in UE yet so if someone can point me in the right direct then that would be great

Cheers

[Stuartf]

Hi

Ive been fiddling but cant get it to work heres what ive done soo far on a 3million line test file

InsertMode
ColumnModeOff
HexOff
UnixReOff
GotoLine 1000000
SelectToBottom
Delete
SaveAs "1-1000000.txt"
CloseFile
Open "Test.txt"
GotoLine 1000000
SelectToTop
Delete
GotoLine 1000000
SelectToBottom
Delete
SaveAs "1000000-2000000.txt"
CloseFile
Open "Test.txt"
GotoLine 2000000
SelectToTop
Delete
SaveAs "2000000-3000000.txt"
CloseFile


I've done it this way because my clipboard cant hold much

It runs up until the 1st Open "Test.txt" then falls over

Please help

[Mofi]

Your macro code looks quite good. Because I don't have a 2 GB text file I can only guess what the problem is: the synchronizing process between end of the opening of the extremly large file and continue with next macro command. Maybe the command Top after every Open helps, maybe in a loop with a defined number to create a wait loop.

And hopefully you have choosen also the option Open file without temp file but NO prompt (CAUTION: Edits are permanent, decreases load time for large files) at Configuration - File Handling - Temporary Files which of course you will definitively need here.

Normally it is also good for editing so large files to check the configuration option Disable line numbers at Configuration - Editor Display - Miscellaneous to decrease the loading time but you can't do that because you need the line numbers and so UE has to scan the whole file for CRLF.

[Mofi]

A second suggestion: After every Open command inserting following:

Bottom
GetValue "Enter 0 when bottom of file reached!"
Key BACKSPACE

Maybe this user interaction helps UE.


Third suggestion: Make the split manually by using Edit - Select Range and File - Save Selection As.


Fourth suggestion: Check the configuration option Disable line numbers and use instead of GotoLine the following macro code:

Loop 1000000
Key DOWN ARROW
EndLoop

Very slow but maybe working.

[Stuartf]

Hi Mofi

Thanks for your help

Is there not a SaveSelectionAs "filename.txt" command or work around?


tx

[Mofi]

No, there is no SaveSelectionAs command because in a macro normally the command sequence

Copy
NewFile
Paste
SaveAs "filename"

does the same. But in your case it will fail with hundreds of MBs.

Well, you could create a submacro which does for example following

IfEof
Else
StartSelect
Loop 200
Key PGDN
EndLoop
Copy
EndSelect
PreviousWindow
Paste
NextWindow
EndIf

And in your main macro you have following sequence as often as needed:

NewFile
NextWindow
PlayMacro 500 "Case sensitive name of already existing submacro"
PreviousWindow
SaveAs "filename01.txt"
CloseFile

But I don't understand why you do not use one of the dozens free file splitting tools available on WWW which would make this job much faster than an editor can ever do because the file splitting tool must not show the content of the huge file.

[Stuartf]

Thanks Mofi

yep your right I'll just buy the software or do it manually only takes about 10 minutes but i thought it would be interesting to see if it was possible in UE.

Thanks for your help


Stu