Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Go Back  Xtreme Visual Basic Talk > > > Loading an immense string taken from an aspx file to a datagridview for manipulation


Reply
 
Thread Tools Display Modes
  #1  
Old 07-23-2008, 08:03 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Question Loading an immense string taken from an aspx file to a datagridview for manipulation


Hi all,

Firstly I would like to thank the people who responded to my last post about Access and .NET it has been really usefull, I just prefer not to to bump the thread.

...

I have another request from my boss. We need to make available the records of an sql table that has thousands of records in a local app for UPDATEs and INSERTs. This app will be used by one of our clients to modify values from random records.

My boss told me about how excel can handle an incredible amount of records and how he was confident a Datagridview object could handle the task. Say the table has something like 80 000 rows; I generated an aspx file with the following synthax: <p>MyVariable: Value</p> ... till last record.

Then with my VB NET app I copied the whole shabang into a string. I then manipulate this string to get each record and populate the datagridview.

Now here's the obvious problem.... it's slower than a snail driving a hyundai in first gear on a sunday with wind coming from the opposite direction.

here are parts of my code:

StringHTML = StringHTML.Substring(StringHTML.IndexOf(":") + 1)
StringNoSeq = StringHTML.Substring(0, StringHTML.IndexOf("<"))

This cuts the string where my next record is then parses it (this might be very slow since the string is immense) Any ideas guys to make this run faster?

Also, after a line of values is parsed, I use this to fill the Datagridview object:

DataGridView.Rows.Insert(0, StringNoSeq, StringNoCOOP, StringCodeProduit, StringDescription, StringFormat, StringFormatSeq, StringFamille)

Is this slowing things? anything better to make it faster?

I really hope you guys can give me a little help with this... you always do, thanks and thanks again.

Cya

Last edited by 7h0m; 07-23-2008 at 01:52 PM.
Reply With Quote
  #2  
Old 07-23-2008, 08:31 AM
Shurik12Loading an immense string taken from an aspx file to a datagridview for manipulation Shurik12 is offline
Steppe Walker

Retired Moderator
* Expert *
 
Join Date: Jul 2002
Location: Ukraine/Russia/Belgium
Posts: 7,227
Default

Hi,

Just to make it sure, are you trying to read in 80.000 records in DataGridView
and displaying the whole thing on an aspx page "at once"?

Regards,
Shurik.
__________________
"A diaper is not like a computer that makes satisfying burbling noises from time to time, hinting at great inner complexity." Malcolm Gladwell

"I'm sitting here completely surrounded by no beer." Onslow, 'Keeping up appearances'
Reply With Quote
  #3  
Old 07-23-2008, 08:40 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Default

I'm just using an ASPX page to generate brute text that i access with my local app using a url.... and yes I am trying to use the huge generated wall of text to populate the datagridview that gives something like 80k records.

Thanks for asking!
Reply With Quote
  #4  
Old 07-23-2008, 09:09 AM
Optikal's Avatar
OptikalLoading an immense string taken from an aspx file to a datagridview for manipulation Optikal is offline
Codeaholic

Retired Leader
* Guru *
 
Join Date: Oct 2002
Location: Winnipeg, MB, Canada
Posts: 4,543
Default

A couple of comments:

1) Why not just have the application with the datagridview connect directly to the database using ADO.Net and retrieve a dataset that you bind the gridview to.

2) Displaying 80k records on the screen at once is pretty pointless. No human is going to scroll through 80k records. It's usually more reasonable to provide the user some search/filter criteria to enter then populate the datagrid based on that. Or use paging to only display a subset of the records at once.
__________________
There are 10 types of people in this world, those that understand binary, and those that don't.
Reply With Quote
  #5  
Old 07-23-2008, 09:17 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Arrow

Quote:
Originally Posted by Optikal View Post
A couple of comments:

1) Why not just have the application with the datagridview connect directly to the database using ADO.Net and retrieve a dataset that you bind the gridview to.

2) Displaying 80k records on the screen at once is pretty pointless. No human is going to scroll through 80k records. It's usually more reasonable to provide the user some search/filter criteria to enter then populate the datagrid based on that. Or use paging to only display a subset of the records at once.
1) This is for security reasons

2) I know... I would make the ASPX page accept a filter criteria but my boss wants her to have the enormous amount of data at all times inside the datagrid. What I would do is repopulate the grid with the search results... but then it would be slower than having them all loaded in one shot, then filter/seacrh, I think that's why he wants to have the data loaded at once (form load)
Reply With Quote
  #6  
Old 07-23-2008, 09:21 AM
darkforcesjedi's Avatar
darkforcesjediLoading an immense string taken from an aspx file to a datagridview for manipulation darkforcesjedi is offline
Trust me, I'm an

* Expert *
 
Join Date: Apr 2001
Location: In ur base, pwnin d00dz
Posts: 1,964
Default

I can definitely see why it's slower than a snail driving a hyundai uphill in first gear on a sunday with a strong headwind.

Using a giant string is part of the problem. And inserting a row into the data grid for each record is going to be slow too. .NET has tools for both generating and parsing XML, which will likely be much faster than what you have.

I'm not sold on the idea of the web server sending you the information in plain text. Even if you do send it in plain text, the HTML formatting serves no purpose since the page is just there to provide information to your application. There's also no need to send the field names with every record.

Edit: As for not being able to connect to the database for security reasons... you're sending the same data in plain text to the user via a web server, how is that more secure?
__________________
To err is human; to debug, divine.
Reply With Quote
  #7  
Old 07-23-2008, 09:42 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Exclamation

Quote:
Originally Posted by darkforcesjedi View Post
I can definitely see why it's slower than a snail driving a hyundai uphill in first gear on a sunday with a strong headwind.

Using a giant string is part of the problem. And inserting a row into the data grid for each record is going to be slow too. .NET has tools for both generating and parsing XML, which will likely be much faster than what you have.

I'm not sold on the idea of the web server sending you the information in plain text. Even if you do send it in plain text, the HTML formatting serves no purpose since the page is just there to provide information to your application. There's also no need to send the field names with every record.

Edit: As for not being able to connect to the database for security reasons... you're sending the same data in plain text to the user via a web server, how is that more secure?
Could you elaborate on these XML tools? How could I generate an XML file from the aspx file to parse it in my datagridview?

I know the HTML formatting doesn't do anything, it's just there for show. I might remove it somewhere down the line if I keep this method.

As for the security measures, it's not the data itself but we don't want them to directly access the database from their servers.
Reply With Quote
  #8  
Old 07-23-2008, 11:10 AM
darkforcesjedi's Avatar
darkforcesjediLoading an immense string taken from an aspx file to a datagridview for manipulation darkforcesjedi is offline
Trust me, I'm an

* Expert *
 
Join Date: Apr 2001
Location: In ur base, pwnin d00dz
Posts: 1,964
Default

The XML tools are in the System.XML namespace. Look at the last example on this page: http://www.beansoftware.com/ASP.NET-...ng-VB.NET.aspx

It makes the web server send an XML document containing the results of the query. You don't have to do any of the work to write the XML file. The XMLTextReader can grab the XML file from the server for you:

http://developer.yahoo.com/dotnet/howto-xml_vb.html
__________________
To err is human; to debug, divine.
Reply With Quote
  #9  
Old 07-23-2008, 11:14 AM
Optikal's Avatar
OptikalLoading an immense string taken from an aspx file to a datagridview for manipulation Optikal is offline
Codeaholic

Retired Leader
* Guru *
 
Join Date: Oct 2002
Location: Winnipeg, MB, Canada
Posts: 4,543
Default

I'm not sure if this is the same as darkforcesjedi suggestion or not, but if you have a valid security concern about exposing your server to the client app, instead of exposing the data via an aspx page why not use a web service (ASMX) instead that exposes a method that returns a Dataset (serialized to XML). Your client app could call the webservice to retrieve a dataset then bind directly to it.
__________________
There are 10 types of people in this world, those that understand binary, and those that don't.
Reply With Quote
  #10  
Old 07-23-2008, 11:31 AM
darkforcesjedi's Avatar
darkforcesjediLoading an immense string taken from an aspx file to a datagridview for manipulation darkforcesjedi is offline
Trust me, I'm an

* Expert *
 
Join Date: Apr 2001
Location: In ur base, pwnin d00dz
Posts: 1,964
Default

Not quite what I suggested, but the same concept (without the web service). I don't really know anything about ASP.NET.

My idea was that the web server send an XML response using DataSet.WriteXML(Response.OutputStream), then the VB.NET client would read that stream either into a DataSet or another location.

If bandwisth is low between the client and web server, you may want to enable HTTP compression for that site (or page) in IIS, since the resulting output should be highly compressible.
__________________
To err is human; to debug, divine.
Reply With Quote
  #11  
Old 07-23-2008, 12:46 PM
Optikal's Avatar
OptikalLoading an immense string taken from an aspx file to a datagridview for manipulation Optikal is offline
Codeaholic

Retired Leader
* Guru *
 
Join Date: Oct 2002
Location: Winnipeg, MB, Canada
Posts: 4,543
Default

If you use a web service it accomplishes the exact same thing but handles all the plumbing for you. Just create a method in the web service project that returns a dataset and mark it with the WebMethod attribute. ASP.Net will automatically turn that into an XML based web-service that serializes the DataSet to XML and exposes it via SOAP.

Then in the client you just add a Web Reference and it will automatically generate a proxy class that handles the plumbing of calling the web service, crafting the SOAP message, and deserializing the result back into a DataSet.

That may sound complex, but the way it's implemented in .Net all that complexity is hidden from you, all you need to do is toss a WebMethod attribute on your server-side method, and add a web reference to the client-side project.
__________________
There are 10 types of people in this world, those that understand binary, and those that don't.
Reply With Quote
  #12  
Old 07-23-2008, 01:51 PM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Arrow

Sorry if I'm dodging some of your suggestions here but I decided to quickly convert the output aspx file into a xml document (server side). Now I'm just trying to parse the string i have containing the data (string) into an Xmldocument object in VB.

The aspx page is protected (https) here is the code that retrieves the XML data...

Dim XmlDataDocument As XmlDocument = New XmlDataDocument()
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705")
client.Credentials = New NetworkCredential("user1", "passme")
StringHTML = New System.Text.UTF8Encoding().GetString(client.DownloadData("http://site/guylaine/OUT.aspx"))

Now that I have the String loaded I'm trying to convert it into the Xml Object:

XmlDataDocument.LoadXml(StringHTML)

It always fails - telling me all sorts of exceptions regarding the type of some chars. As if there were characters getting droppped (during the UTF-8 formatting?). So I tried some hacking switching chars with this code:

While v <= 48
StringHTML = StringHTML.Replace(ChrW(v), "")
v = v + 1
End While

it finally resulted in the error:

"Data at the root level is invalid. Line 1, Position 1."

I then tried as suggested elsewhere to use Load and not LoadXML; that would correct the said error.

XmlDocument.Load(StringHTML)

...but it seems the 'string' is too big. That's the error I'm getting.

Well I know it's kinda messed up but if I could get the String from the server correclty and parse it into the XML object I'd be done with it all.

Any ideas on the characters being scrambled? Or the whole String to XML convertion?

Thanks a-lllot. I'm really looking forward to your replies.
Reply With Quote
  #13  
Old 07-23-2008, 02:22 PM
Optikal's Avatar
OptikalLoading an immense string taken from an aspx file to a datagridview for manipulation Optikal is offline
Codeaholic

Retired Leader
* Guru *
 
Join Date: Oct 2002
Location: Winnipeg, MB, Canada
Posts: 4,543
Default

You have to make sure the string your retrieving is a valid xml document.

That means it has the <?xml...> stuff at the start, and is well-formed.

Are you making sure your not including the <html><head> etc stuff in the string your passing to the xml classes?
__________________
There are 10 types of people in this world, those that understand binary, and those that don't.
Reply With Quote
  #14  
Old 07-24-2008, 08:31 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Default

I think my problem now is that some characters in the data retrieved from the database contains '<' and '>' around the place. I should perhaps catch them in my ASP file but I'm afraid that will slow down the code on the server side and modify the records in ways I don't really want.

Just letting you guys know I'm here and still trying to parse a XML file to the datagridview.
__________________
Nasa predicts Pole Shift in 2012. Google it.
Reply With Quote
  #15  
Old 07-24-2008, 08:52 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Default

What would be another alternative/structure that the Datagridview object could use as a Datasource? Or is there any simple way to dig out the wrong chars in my records with ASP without taking too much cpu time? Here's my asp code:

<%
Response.Write("<?xml version='1.0' encoding='utf-8'?>")
rs.Open ( "EXEC USP_LIST_PRODUIT", ConnectionSQL)
Response.Write("<Liste_Produit>")
While (Not rs.EOF)
Response.Write("<Produit>")
Response.Write("<Seq>" & rs.Fields("Seq").Value & "</Seq>")
Response.Write("<Coop>" & rs.Fields("Coop").Value & "</Coop>")
Response.Write("<Code>" & rs.Fields("Code").Value & "</Code>")
Response.Write("<Nom>" & rs.Fields("Nom").Value & "</Nom>")
Response.Write("<FormatListe>" & rs.Fields("FormatListe").Value & "</FormatListe>")
Response.Write("<FormatSeq>" & rs.Fields("FormatSeq").Value & "</FormatSeq>")
Response.Write("<Famille>" & rs.Fields("Famille").Value & "</Famille>")
Response.Write("</Produit>")
rs.MoveNext()
end while
rs.Close()
Response.Write("</Liste_Produit>")
%>

Thanks for sticking up.
__________________
Nasa predicts Pole Shift in 2012. Google it.
Reply With Quote
  #16  
Old 07-24-2008, 09:22 AM
darkforcesjedi's Avatar
darkforcesjediLoading an immense string taken from an aspx file to a datagridview for manipulation darkforcesjedi is offline
Trust me, I'm an

* Expert *
 
Join Date: Apr 2001
Location: In ur base, pwnin d00dz
Posts: 1,964
Default

Are you using classic VBScript ASP, or ASP.NET? I thought you mentioned using ASP.NET.

In one of the links I posted, there is code to serialize a dataset (similar to an ADO recordset):
Code:
        Dim ds As New dataset
        Dim connStr As String = "database=NorthWind;Data Source=localhost;User id=sa;password=sa"
        Using conn As New SqlConnection(connStr)
            Dim command As New SqlCommand("select * from customers", conn)
            conn.Open()
            ds.DataSetName = "Customers"
            ds.Load(command.ExecuteReader(), LoadOption.OverwriteChanges, "customers")
            Response.ContentType = "text/xml"
            ds.WriteXml(Response.OutputStream)
        End Using
__________________
To err is human; to debug, divine.
Reply With Quote
  #17  
Old 07-24-2008, 09:42 AM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Default

I'll try to modify the code to make it work since I'm using a Stored Proc with Recordset..
__________________
Nasa predicts Pole Shift in 2012. Google it.
Reply With Quote
  #18  
Old 07-24-2008, 12:44 PM
Optikal's Avatar
OptikalLoading an immense string taken from an aspx file to a datagridview for manipulation Optikal is offline
Codeaholic

Retired Leader
* Guru *
 
Join Date: Oct 2002
Location: Winnipeg, MB, Canada
Posts: 4,543
Default

Quote:
Originally Posted by 7h0m View Post
I'll try to modify the code to make it work since I'm using a Stored Proc with Recordset..
I think the confusion is you said you were using ASP.Net (and posted in the .Net forum) but your code doesn't look like .Net. What type of object is rs???
__________________
There are 10 types of people in this world, those that understand binary, and those that don't.
Reply With Quote
  #19  
Old 07-24-2008, 12:57 PM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Default

sorry this is ASP.NET

rs is a record set
__________________
Nasa predicts Pole Shift in 2012. Google it.
Reply With Quote
  #20  
Old 07-24-2008, 01:19 PM
7h0m's Avatar
7h0m 7h0m is offline
Regular
 
Join Date: Feb 2008
Posts: 64
Default

Here is my updated code: (aspx)

<%
Response.Write("<?xml version=""1.0"" encoding=""UTF-8"" ?>")
rs.Open ( "EXEC USP_LIST_PRODUIT", ConnectionSQL)
Response.Write("<Liste_Produit>")
While (Not rs.EOF)
Response.Write("<Produit>")
Response.Write("<Seq>" & RChars(rs.Fields("Seq").Value) & "</Seq>")
Response.Write("<Coop>" & RChars(rs.Fields("Coop").Value) & "</Coop>")
Response.Write("<Code>" & RChars(rs.Fields("Code").Value) & "</Code>")
Response.Write("<Nom>" & RChars(rs.Fields("Nom").Value) & "</Nom>")
Response.Write("<FormatListe>" & RChars(rs.Fields("FormatListe").Value) & "</FormatListe>")
Response.Write("<FormatSeq>" & RChars(rs.Fields("FormatSeq").Value) & "</FormatSeq>")
Response.Write("<Famille>" & RChars(rs.Fields("Famille").Value) & "</Famille>")
Response.Write("</Produit>")
rs.MoveNext()
end while
rs.Close()
Response.Write("</Liste_Produit>")

%>

<script runat="server" language="vb">

Function RChars(ByVal Field As String)

Field = Replace(Field, "&", "&amp;")
Field = Replace(Field, "<", "&lt;")
Field = Replace(Field, ">", "&gt;")
Field = Replace(Field, """", "&quot;")
Field = Replace(Field, "'", "&apos;")
Field = Replace(Field, ChrW(16), "")

return Field

end function

</script>

But again, it seems I will end up with broken records if I switch every characters that are not displayed. They appear as errors in VB NET in square chars form. perhaps changing the encoding might work?
__________________
Nasa predicts Pole Shift in 2012. Google it.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Advertisement:





Free Publications
The ASP.NET 2.0 Anthology
101 Essential Tips, Tricks & Hacks - Free 156 Page Preview. Learn the most practical features and best approaches for ASP.NET.
subscribe
Programmers Heaven C# School Book -Free 338 Page eBook
The Programmers Heaven C# School book covers the .NET framework and the C# language.
subscribe
Build Your Own ASP.NET 3.5 Web Site Using C# & VB, 3rd Edition - Free 219 Page Preview!
This comprehensive step-by-step guide will help get your database-driven ASP.NET web site up and running in no time..
subscribe
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
 
Loading an immense string taken from an aspx file to a datagridview for manipulation
Loading an immense string taken from an aspx file to a datagridview for manipulation
 
-->