Huge Datamining = Huge DB - How to get best Performance etc?

[Exxtromo]

Hey!

I bought hand histories from ipoker from the last 2 month (about 6 million hands). I am scanning them at the moment. After 1 Mio hand the Housekeeping takes about 25 minutes, so i guess it will take much longer after i have scanned my 6 mio. I also will have every single hand played on my 2 limits i play in the future...so i will have a HUUUGE database.

My question is, does it really make sense to use only one database or should i create a new one every month or every 2 months? Does it make sense to Housekeep my database every few days?
Dos it make a difference if i buy a better CPU? (i now use 2x2,33Ghz with 2 Gig Ram with a good setup)?

I want to get the best performance i can get with this huge and fast growing database. Hope u can give me some tipps to do so.

Thanks a lot for ur help!

[WhiteRider]

Check out the 'general' post in the Database / Postgres FAQ

[Exxtromo]

well okay. Im still scanning. The scan work is pretty sick.

I will create ONE database for every limit every 2 weeks. Thats a lot of work, but Housekeeping one month of hands of one or two limits takes soooo much time (couple of hours) .

I think i will have to find it out for myself whats the maximum size of a database.

Pretty hard work anyway....this really sucks ;)

[WhiteRider]

I don't think you need to break it down quite that much.
1 database per month for all levels should be enough - or even every couple of months, 6 million hands should be ok. Once you have a 'full' DB (i.e. you're not going to import into it again) and run all the housekeeping on it you won't need to do it again because if you don't update it it will stay in optimum condition.

(BTW, I'll move this to the PT3 section)

[Exxtromo]

thanks for the advice. i will make one database per month...thats easy to manage

last question: when i import one or 2 million hands in a row, does it make sense to cluster/vacuum/analyse/cache after a while, or is it enough to do it at the end?

[WhiteRider]

No, just do it at the end - doing it part way through is a waste of time because it will all be redone at the end.

[Exxtromo]

thank you very much

[kaisersoze]

Hi,

I had 2 databases : 1 for played hands and 1 for observed hands.

My DB on observed hands has 1,5 Million hands (approx.). When i cluster or update the cache it takes 1H30mn for each. So i decided to create a 2nd database for observed hands to have housekeeping faster.
It's a good idea ?

For the hud, i check the 3. It will merge from the 3 if i understood good.

I have intention to create a new DB every 1 Mhands.

For the Hud in the replayer (future update), it will show stats merged from the 3 databases ?

[kraada]

Yes the HUD will show data from all three databases if you check them in the HUD Options; I can't say for certain how the HUD in the replayer will display things but I strongly suspect that it will work the same there as well.

Creating a new database every million hands will make things significantly easier yes; once you've run housekeeping on a database you no longer intend to use, you don't need to re-run it again which will save you quite a bit of time after a while.

[kaisersoze]

Thanks.

Do you know if the Hud in the replayer will be in the next beta ?