Fastest way to read text file
Fastest way to read text file
Fastest way to read text file
Fastest way to read text file
Fastest way to read text file
Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file
Fastest way to read text file Fastest way to read text file
Fastest way to read text file
Go Back  Xtreme Visual Basic Talk > > > Fastest way to read text file


Reply
 
Thread Tools Display Modes
  #1  
Old 02-05-2009, 12:21 PM
Greatchap Greatchap is offline
Regular
 
Join Date: Feb 2003
Location: India
Posts: 87
Question Fastest way to read text file


Hi Guys,

I have a coma seperated text file with loads of records. Usually when we read a text file we start from the first line and go towards the last. I know that there will be x comas per record.

But I want to start reading from the last line and then towards the top. This enables me to stop before I reach the top (depending on the data).

What would be the fastest way to read a text file in VB.NET. I am using VS2008. Basically I want to read from last line and stop when I want. I may not need to read the entire file.

Valuable suggestions would be appreciated.

Thanks,
GR
Reply With Quote
  #2  
Old 02-05-2009, 01:14 PM
whlabus whlabus is offline
Freshman
 
Join Date: Mar 2007
Posts: 29
Default

What type of data are you storing? I would highly recommend using something like an XML file. You'll find they're incredibly versatile as well as easy to work with in Visual Basic. Plain text files are not a very practical or efficient way to store data, for the reason you described- pretty much the only way to read and modify them is in a linear approach.

If you give some info on the kind of data you're storing, I could get you started with using XML instead. It really is very easy and practical.
Reply With Quote
  #3  
Old 02-05-2009, 01:45 PM
AtmaWeapon's Avatar
AtmaWeaponFastest way to read text file AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

An XML file's going to be easier to manage in this fashion, but if the file is rather large this is going to use memory much worse than a CSV file since DOM readers have to load the entire document into memory.

You can use random access to read a line, then attempt to go backwards a line in the file, but it's probably going to be really messy. You can only move backwards a specific number of bytes, so you'd have to either have a predictable fixed line size or do a lot of experimentation.

I'm not really sure there's a decent solution other than to change how the file is structured so that the "last" line is first.
__________________
.NET Resources
My FAQ threads | Tutor's Corner | Code Library
I would bet money 2/3 of .NET questions are already answered in one of these three places.
Reply With Quote
  #4  
Old 02-05-2009, 01:48 PM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

Quote:
Originally Posted by whlabus View Post
What type of data are you storing? I would highly recommend using something like an XML file. You'll find they're incredibly versatile as well as easy to work with in Visual Basic. Plain text files are not a very practical or efficient way to store data, for the reason you described- pretty much the only way to read and modify them is in a linear approach.
XML files *ARE* text files.

I think that you have confused a rich support library for XML files with "features."

What happens when you move to a language or platform without that rich support library? XML files become DOA because it is non-trivial to re-implement.

Also, what about XML makes reading data backwards efficient?

Quote:
Originally Posted by whlabus View Post
If you give some info on the kind of data you're storing, I could get you started with using XML instead. It really is very easy and practical.
CSV files are easy, practical, and are supported as standard by many database and spreadsheet applications. Most scientific datasets that you will find online are in CSV (climate records, interstellar x-ray surveys, etc..) CSV, essentialy, is universal. XML isnt well supported in packages like R (the defacto standard statistics package)
Reply With Quote
  #5  
Old 02-05-2009, 01:51 PM
Rockoon's Avatar
Rockoon Rockoon is offline
Joseph Koss

* Guru *
 
Join Date: Aug 2003
Location: Unfashionable End
Posts: 3,615
Default

If the entire file will fit into memory, then it should be pretty efficient to read it in as one long string, and the Split() it at line-breaks.

If the file cannot be held in memory (some CSV datasets can be HUGE, I know..), then creating an index table which simply contains offsets to the start of each line, should offer relatively fast reverse iteration. Save this index to disk so that it only has to be indexed once.
Reply With Quote
  #6  
Old 02-05-2009, 03:40 PM
AtmaWeapon's Avatar
AtmaWeaponFastest way to read text file AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

Quote:
Originally Posted by Rockoon View Post
If the entire file will fit into memory, then it should be pretty efficient to read it in as one long string, and the Split() it at line-breaks.

If the file cannot be held in memory (some CSV datasets can be HUGE, I know..), then creating an index table which simply contains offsets to the start of each line, should offer relatively fast reverse iteration. Save this index to disk so that it only has to be indexed once.
I like this answer; it should be relatively easy to generate this index table, and it lets you use random access to work with one line of the file at a time.
Reply With Quote
  #7  
Old 02-05-2009, 10:30 PM
Greatchap Greatchap is offline
Regular
 
Join Date: Feb 2003
Location: India
Posts: 87
Red face txt files

The data in my txt files is like this -

01/02/2009,1050,123,124,122,121,34567
01/02/2009,1055,1230,1240,1220,1210,345679
01/02/2009,1100,23,24,22,21,679
..... so on

Now if there are large records then I dont want to read from start (begining of file to end) as it takes longer. I want to start from last record.

currently I am using a .net code like IO.File.ReadAllLines("PATH") and it reads entire file in an array. The I read the array from bottom to top.

Is this method efficient or is there a faster way ??
Reply With Quote
  #8  
Old 02-06-2009, 10:27 AM
AtmaWeapon's Avatar
AtmaWeaponFastest way to read text file AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

Read posts 3-6.

If you disagree with the viewpoints or if you have questions about them, it's quite impolite to ignore them and pretend they don't exist. It'd be much better to ask questions or explain why you think the approach won't work.

Ignoring the posts above makes me feel like you're looking less for an answer and more for someone to write something you can copy and paste into your application. I don't like to give that kind of help, because it doesn't really teach anything.
__________________
.NET Resources
My FAQ threads | Tutor's Corner | Code Library
I would bet money 2/3 of .NET questions are already answered in one of these three places.
Reply With Quote
  #9  
Old 02-06-2009, 11:16 PM
Greatchap Greatchap is offline
Regular
 
Join Date: Feb 2003
Location: India
Posts: 87
Unhappy Apologies

I am sorry. I think there has been a misunderstanding. I am grateful to you guys as you reply to my posts and help me out. There is no way I would ignore any post.

The only reason I wrote the structure of my text file is because in the 2nd post I was asked to mention what type of data is present in the files. Moreover my mistake as I didnt mention that the structure of text files cannot be changed so I cannot create XML files and move on. Thats why I asked if there is any other way than xml or spliting the data after its read in an array.

P.S. I am not looking for someone to write code. I just need some help as to what method should I adopt.

I am sorry if I have been impolite.

Thanks a lot for your support.
Reply With Quote
  #10  
Old 04-16-2009, 03:15 PM
Ignyte_Software Ignyte_Software is offline
Newcomer
 
Join Date: Apr 2009
Posts: 1
Default StreamReader appears to be faster than IO.File.ReadAllLines

Greatchap,

I have tried using both of the following:

1. IO.File.ReadAllLines("PATH") - your current approach

2. Using StreamReader

objStreamReader = New StreamReader(path)
strline = objStreamReader.ReadLine

In my testing, StreamReader has been twice as fast as IO.File.ReadAllLines("PATH"). This was proved during testing that read in more than 6 million records from within 30 different text files.

Quote:
Originally Posted by Greatchap View Post
The data in my txt files is like this -

01/02/2009,1050,123,124,122,121,34567
01/02/2009,1055,1230,1240,1220,1210,345679
01/02/2009,1100,23,24,22,21,679
..... so on

Now if there are large records then I dont want to read from start (begining of file to end) as it takes longer. I want to start from last record.

currently I am using a .net code like IO.File.ReadAllLines("PATH") and it reads entire file in an array. The I read the array from bottom to top.

Is this method efficient or is there a faster way ??
Reply With Quote
  #11  
Old 04-17-2009, 08:58 AM
Greatchap Greatchap is offline
Regular
 
Join Date: Feb 2003
Location: India
Posts: 87
Default

Thanks for the information. I get it, as I am reading files whose sizes are a couple of mbs I guess I will stick to ReadllLines.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Advertisement:





Free Publications
The ASP.NET 2.0 Anthology
101 Essential Tips, Tricks & Hacks - Free 156 Page Preview. Learn the most practical features and best approaches for ASP.NET.
subscribe
Programmers Heaven C# School Book -Free 338 Page eBook
The Programmers Heaven C# School book covers the .NET framework and the C# language.
subscribe
Build Your Own ASP.NET 3.5 Web Site Using C# & VB, 3rd Edition - Free 219 Page Preview!
This comprehensive step-by-step guide will help get your database-driven ASP.NET web site up and running in no time..
subscribe
Fastest way to read text file
Fastest way to read text file
Fastest way to read text file Fastest way to read text file
Fastest way to read text file
Fastest way to read text file
Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file Fastest way to read text file
Fastest way to read text file
Fastest way to read text file
 
Fastest way to read text file
Fastest way to read text file
 
-->