Go Back  Xtreme Visual Basic Talk > Visual Basic .NET (2002/2003/2005/2008, including Express editions) > .NET General > Fastest way to load text file into ListBox/ArrayList


Reply
 
Thread Tools Display Modes
  #1  
Old 04-30-2010, 12:07 AM
Jellio Jellio is offline
Regular
 
Join Date: Nov 2009
Posts: 56
Default Fastest way to load text file into ListBox/ArrayList


I would like to know if there is a quicker way to load a text file into a listbox/arraylist than this:

Code:
Dim fileList as New ArrayList
Dim fileReader As New System.IO.StreamReader("textfile.txt")
Do While fileReader.Peek() <> -1
   fileList.Add(fileReader.ReadLine())
Loop
fileReader.Close()

If I need to load several text files into different arraylists, especially if the textfiles have a lot of lines, this process is very slow. Too slow for my current app to work smoothly, actually.

So, is there any faster ways to load a text file into an arraylist? Or if there isn't, is there some quick way to easily access and change data that is on a particular line in a textbox, so that I don't have to load the text file into an arraylist?
Reply With Quote
  #2  
Old 04-30-2010, 08:35 AM
AtmaWeapon's Avatar
AtmaWeapon AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

First, don't use ArrayList unless you're using Visual Studio 2003. If you are, you really need to upgrade. ArrayList has several issues and you should prefer using the List(Of T) class in .NET 2.0 and beyond. There's a discussion in this thread that explains more in detail; I don't want to waste time repeating it.

Now to actually discuss the question.

There's probably not a way to speed up the process. File I/O is slow. Your computer views the world in nanoseconds. Data arrives from the hard drive in millisecond intervals. Let's put it in human terms. If 1 second is one clock cycle, accessing data on a hard drive generally takes 5,000 seconds (1.4 hours!). That's how slow the hard drive is, and 5ms represents a pretty good hard drive. So you're not going to easily get data out of the file in a hurry.

So you'll have to resort to clever tricks and worker threads.

You might handle the "app doesn't run smoothly" problem by loading the data in a worker thread. Be aware that you can't directly work with a control from a worker thread; I've explained some solutions in the past. It'll still take a while to load the data, but the program won't freeze while it's being loaded.

If you're loading that many lines into a ListBox, you're going to see some slowdown anyway. You might speed up the insertion of data using the BeginUpdate() and EndUpdate() methods, but in my experience a list box with thousands of lines just doesn't work well. You might consider using a ListView instead; when you operate it in "virtual" mode it only loads the data it is displaying into memory. I've explained how it works before; I swear I've written an example but I can't find it.

I could go on, but you'd get the best advice if you'd describe what you're doing instead of how you're doing it.
__________________
.NET Resources
My FAQ threads | Tutor's Corner | Code Library
I would bet money 2/3 of .NET questions are already answered in one of these three places.
Reply With Quote
  #3  
Old 04-30-2010, 02:15 PM
Jellio Jellio is offline
Regular
 
Join Date: Nov 2009
Posts: 56
Default

Quote:
I could go on, but you'd get the best advice if you'd describe what you're doing instead of how you're doing it.
I'm working on a football(american) management game, and the various data(player attributes, team attributes, schedules etc. etc.) are currently loaded in ArrayLists. Each player has an ID number, so that player no.165's attributes will be in playerattributeList(165) and player no. 543's attributes will be in playerattributeList(543) etc. Specific attributes are on specific 'lines', i.e the players first name is item no. 12 and the players last name is item no. 13 etc.

At first, making the game was just a "hobby", as I'm not really a programmer and wasn't sure I would ever know how to actually do it, and that's why I probably didn't do enough planning when I started. The current solution to saving/loading the game, saving the contents of these ArrayLists to regular text files, worked well for the longest time, but as the game has grown bigger and bigger, the saving, and especially the loading process takes longer and longer. It already takes too long for it to be enjoyable to play, and that's why I probably have to do the thing I didn't want to do when I started doing this game: Start learing how to use databases.
This thread was just a final attempt to maybe fasten the loading process(which is the bigger problem) while using this system, but I guess I'll have to turn to databases, because even though the game itself runs really smoothly, the loading and saving just takes too darn long.

So, if you have some good links to some easy tutorials on using i.e Access databases in WPF, I'd be interested. Ideally I'd want to make the app load specific datatables(or whatever they are called) into the ArrayLists, so that I wouldn't have to change too many things about the actual game, but I don't know if that actually is much faster. I am a complete beginner in terms of databases, so I really don't know what the best solution is.
Reply With Quote
  #4  
Old 04-30-2010, 02:47 PM
AtmaWeapon's Avatar
AtmaWeapon AtmaWeapon is offline
Fabulous Florist

Forum Leader
* Guru *
 
Join Date: Feb 2004
Location: Austin, TX
Posts: 9,500
Default

Again, don't use ArrayLists. They aren't type-safe and they cause boxing/unboxing. The first one causes errors. The second causes performance problems:
Code:
Dim iterations As Integer = 1000000

Dim profiler As New Stopwatch()

profiler.Start()
Dim list As New List(Of Integer)()
For i As Integer = 0 To iterations - 1
    list.Add(i)
Next

For i As Integer = 0 To iterations - 1
    Dim value As Integer = list(i)
Next
profiler.Stop()
Console.WriteLine("List(Of T): {0:F2}ms", profiler.Elapsed.TotalMilliseconds)

profiler.Restart()
Dim aList As New ArrayList()
For i As Integer = 0 To iterations - 1
    aList.Add(i)
Next

For i As Integer = 0 To iterations - 1
    Dim value As Integer = CInt(aList(i))
Next
profiler.Stop()
Console.WriteLine("ArrayList: {0:F2}ms", profiler.Elapsed.TotalMilliseconds)
On my machine, it's an average of 30ms for List(Of T) and 140ms for ArrayList. List(Of T) is almost 5 times faster. ArrayList is for people using .NET 1.1 and stuck in VS 2003. Don't use it if you're not one of those poor souls. (It's over 1,000,000 iterations so it's mostly moot, but why choose the inferior option when a much better one is available?)

Other than that, I'm not much help. I don't write database code much. I know it would definitely help with loading portions of the data quickly, but I'm not convinced you need them. Have you done profiling and proved that the parts of your application that are slow are the parts that load data? Have you thought about alternative ways to store the data that might be faster? How many players are there and how many attributes do you store per player? What counts as "too slow"? It shouldn't take more than a second or so to load a couple thousand lines from a file and that seems sufficient.
__________________
.NET Resources
My FAQ threads | Tutor's Corner | Code Library
I would bet money 2/3 of .NET questions are already answered in one of these three places.
Reply With Quote
  #5  
Old 04-30-2010, 11:43 PM
Jellio Jellio is offline
Regular
 
Join Date: Nov 2009
Posts: 56
Default

Quote:
Originally Posted by AtmaWeapon View Post
Other than that, I'm not much help. I don't write database code much. I know it would definitely help with loading portions of the data quickly, but I'm not convinced you need them. Have you done profiling and proved that the parts of your application that are slow are the parts that load data? Have you thought about alternative ways to store the data that might be faster? How many players are there and how many attributes do you store per player? What counts as "too slow"? It shouldn't take more than a second or so to load a couple thousand lines from a file and that seems sufficient.
My computer is pretty slow, but it takes almost 15 minutes to load a game currently. It's not the actual loading of the file that takes time, but instead it's the line-by-line adding of items by the StreamReader. There are currently about 4000 players in my database, and each player has about 500 attributes/other data.

I'll try changing the ArrayLists to List(Of T)'s, and see if that makes any difference.
Reply With Quote
  #6  
Old 05-01-2010, 02:02 AM
Jellio Jellio is offline
Regular
 
Join Date: Nov 2009
Posts: 56
Default

I changed to ArrayLists to List(Of T)'s, and it did make the actual game much smoother, and the processing times to advance a day forward in my game became faster, so thank you for that. It did not help all that much(maybe a minute or two) in terms of the problem I have of loading the text files onto the lists, and I think it's the StreamReader that is at the heart of the problem.

I got this idea, but wasn't sure how to do it, or if it would actually be any faster. If I just loaded the text files to a TextBox first(because that is really fast), and then split the TextBox.Text, so each line would be separate, and then would transfer them to a List(which should be faster than the StreamReader). The only problem is, I don't know how to split the TextBox.Text so that each line is separate. I tried this:

Code:
Dim mySplitter As String() = myTextBox.Text.Split(vbNewLine)
but it didn't work as I hoped. It did split the text at the line breaks, but the strings had these weird symbols after them, and I don't know how to get rid of them. When I tried adding each of these mySplitter strings to a ListBox, the ListBox items did show the correct strings, but they also had a line break after them. It is like it did split the text at the correct spot, but still included the line break in the string. So, how do I split the text at the line break, while also removing the line break from the string.



EDIT: I figured out how to split the text at the line breaks. I found the solution here: http://www.xtremevbtalk.com/showpost...64&postcount=3

Code:
Dim Arry() As String = {vbNewLine}
txtLines = sIn.Split(Arry, StringSplitOptions.None)

Last edited by Jellio; 05-01-2010 at 02:09 AM.
Reply With Quote
  #7  
Old 05-02-2010, 07:44 AM
Jellio Jellio is offline
Regular
 
Join Date: Nov 2009
Posts: 56
Default

Before I move on to learning how to use databases, I have one final question:

I noticed that it wasn't the streamreader that was at fault, and that it's simply the massive amount of loading text-files that slows my load process down. Thats why I would like to know which is faster:

a) loading 3000 separate files with 500 lines

or

b) loading 500 separate files with 3000 lines

If the answer is b, I could still rework the text files I have, so that each file would hold 10 or 20 different players info. That way, I wouldn't need to load that many files.

I wouldn't want to do that before I know if it's any faster, because reworking those files would take some time. So, if someone knows whether b is faster than a, I'd love to hear it....
Reply With Quote
  #8  
Old 05-02-2010, 12:10 PM
Jellio Jellio is offline
Regular
 
Join Date: Nov 2009
Posts: 56
Default

I guess I have found my solution, as I did some tests with those load times.

It is way faster to have a few longer files, than many shorter files. I did tests with

A) 4000 files that held 2000 lines each.
B) 250 files that held 32000 lines each.

A took about 50secs and B took about 24secs, during the first load. Once the files were in cache, A still took 6secs or more each time, while B took about a second or even less.

I'll guess I'll start working on implementing this in my game, because if I could cut the load times in half it would be more than adequate.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Advertisement:





Free Publications
The ASP.NET 2.0 Anthology
101 Essential Tips, Tricks & Hacks - Free 156 Page Preview. Learn the most practical features and best approaches for ASP.NET.
subscribe
Programmers Heaven C# School Book -Free 338 Page eBook
The Programmers Heaven C# School book covers the .NET framework and the C# language.
subscribe
Build Your Own ASP.NET 3.5 Web Site Using C# & VB, 3rd Edition - Free 219 Page Preview!
This comprehensive step-by-step guide will help get your database-driven ASP.NET web site up and running in no time..
subscribe
 
 
-->