RF Generation Message Board

So, I own quite a few games digitally, either through Steam, the PSN, the Nintendo eShop, or just DRM-free on my laptop. Most off them are in the database, and I'm happy about that, but I hate adding them because it asks if I have the box and/or manual for them even though, for obvious reasons, that would be impossible. So far I've just put 0 in both value boxes, but it doesn't feel like I should have to, you know?

I don't know how much trouble it would be to address that sort of thing, but it seems like it would be worth looking into.

On a related note, some new games just don't have manuals anymore, even if you buy a physical copy. And I'm not talking about how some manuals are just two pages nowadays, I mean they literally don't have a manual inside, like Star Trek for the PS3. I'm assuming the 360 version is the same as well.

Moving the digitally distributed games into their own database is something i have been clamoring for since before they even were part of the database as it is now.

Hopefully with enough vocal members it will get done.

I'd take it one step further and allow for direct import from services that offer it. For instance I'd love for my Xbox live games to be included and it wouldn't be too hard to use https://xboxleaders.com/ activity stream to bootstrap it.

That i would be against.

It works great for places like www.trueachievements.com and/or www.truetrophies.com (who i would almost guarantee don't use a program similar to that one) but in an instance of adding games to a collection it doesn't know if someone is playing a digital or retail copy. What region that game is from (barring the stackable achievemen/trophy lists and region specific titles) or if the person actually owns the game or is just borrowing it.

And yes you can still borrow digital games. It is a bit trickier and i believe breaks most if not all companies ToU/ToS but i am not 100% sure.

From my understanding, Microsoft does not want people auto retrieving things like achievements and game lists.  It's too much strain on their systems but it's also really hard to stop that.  Some sites like xbox360achievements.org respect this, but others, like raptr, don't care. 

And I agree with Izret - I would rather enter it in manually to verify that everything is correct.  It would be annoying to force add games that somebody no longer wants on the HD. 

Correct Microsoft (since i am most familair with them) is very much against data scraping. They have it written into at least one of their legal documents that everyone agrees to whe they join xbox live or buy a game or visit their website or whatever else they can use to enter into an agreement with you the consumer.

Some sites have found ways around it and MS lets some sites get away with it. But they have cracked down on some and tend to change things around every year (sometimes more often) in an effort to knock down the sites/programs that do.

In fact the site flyingclimber linked to the guy stopped supporting the program in April of this year and elsewhere posted that he hoped there were no major rewrites to Xbox.com (didn't check the age of the comment.)

Microsoft has a developer program that they allow select websites/developers to use APIs. The problem with TrueAchievements is that they can't get enough information from it, and have to selectively scrape SmartGlass/Xbox.com for the information. As you mentioned, Microsoft "allows" it, but they tend to allow those types of things from approved developers rather than just anyone.

As for "importing", it's something we can do for Steam and other digital distributors, who only have one format, but not for mixed distribution systems like Xbox 360 and PS3.

To address the "digital" needs, we need to balance extended the current system to support it, which takes a decent amount of work, vs. fixing the source of the problem, which takes a significant amount of work. However, fixing the source of the problem eventually needs to be tackled. I'll need to balance the amount of work needed for both and decide what to do here shortly, as we need to address the validity of manuals for certain games/consoles as well.

Honestly, would it be too difficult to allow negative numbers in the database for the box, and manual (or make the field nullable)?  If you see null or anything less than 0 in that field, then the UI should just display N/A.  For me personally, that would be enough.

I don't really fully agree with Izret about splitting it into 2 databases.  When you have search functionality, stats, etc, then you create a world of enormous problems and your performance is going to go to s***.  I know this from experience and generally, splitting the database into multiple databases is a bad idea for the same data. 

Unless you're going to split the UI as well (make digital not part of the current UI at all), then I would just extend the existing database schema to include the fields you need, while adding support for other things.  Right now, for digital, the only problem/issue I see is that they don't have boxes or manuals, and showing N/A in the UI would solve that.

The drop box says "Databases"

I have always wanted Hardware, Software and Digital to each have their own dedicated search/database under that drop box.

Hardware already has a different page layout but too much of it was lifted from software. Digital most in need of a completely different layout is identical to software.

Yeah. I'd lean on not splitting your DB if there isn't a strong scaling reason or functional difference for it. At the end of the day it's another game with a digital or physical property for me. No rush on this as I still prefer physical media but looking at my digital collections I only see thai increasing over time.

i don't entirely understand how multiple databases would be worse for performance. if someone is searching for a digital game, and that search function only had to crawl through digital games, wouldn't that be much faster than crawling through all of physical games, hardware, and digital games? i'm sure it would take extra storage space and some more coding/search pages, but otherwise i don't see the issue.

I highly doubt it would have any effect on our hardware, if anything, it would be like what you're saying and would be slightly faster. The whole problem with a digital database is one of man-power, not computing power. The few people we have that can do something like this are still busy fixing bugs that have arisen over the last decade, there's just very little time to devote to creating entirely new features.
The good news is that the list of bugs is getting shorter and shorter, so eventually, we will be able to devote some real time and effort into getting a digital DB up and running.

You are correct if you want to isolate digital games from physical games.  But I'm talking about having a search query and just getting results back for games from all platforms, digital or physical.  You take a performance hit if you have to hit multiple databases because you can't fully utilize the sql query logic to return results depending on how advanced your search is. 

Why do you want to separate digital games from physical games so badly?  They're all just games.

i think some of the main reasons are:
 - digital games should have their own pages with their own list of information, as a lot of the information currently used doesn't really apply to digital games
 - getting rid of the manual/box listings for them in the collections would be great
 - some people here don't want anything to do with digital games, they're physical snobs ;) so it would be nice to keep them separate
 - general decluttering...it would be nice to have hardware and digital games separate as the current database is already massive

i'm sure NES_Rules or Shadow_Kisuragi could add more or correct me on those but that's all i can think of right now

The main reason I'm in favor of separating them is to allow them to have their own unique set of information fields. Basically, things like box and manual scans would be gone. UPC, media format, etc would be gone, and new fields would be added to give a better picture of what the entry is actually detailing. Plus, we'd be able to track the multitude of digital distribution systems out there, not just the main ones, and possibly be able to actually show what system the game you already have is also compatible with.

Once in your collection, they would essentially look the same though, so to your point "they're all just games" that would still be true, the only real difference would be different info fields and instead of an S for software or an H for hardware, it'd be D for digital.

But the collection lists display the boxes and manuals.  Would I have to go to a separate collection in order to view my digital games vs my physical games?

It would be separated just as hardware is currently separated from software. Ideally, you'll be able to mix them in when/if you want and have them completely separate when/if you want.

Ideally each would be its own folder. (Again IMO)
Then there would not be the issue of having check boxes for a box or manual.

My collection (when fully up to date) is software, hardware, digital, demos(which most people don't split out it seems) orphans(specifically boxes/manuals).

i like the idea of a separate folder for digital, and another for hardware. though i understand those people who would prefer it all in one place. when this eventually does get rolling, would it be difficult to give members the choice of whether they want their digital library integrated (visually at least) with their regular collection folder? that way when you go through the add games process it would automatically add them to the main collection folder, instead of i assume the original idea was to have them automatically add to the digital folder instead.

that being just an option for users to enable or disable, with it disabled by default...default being to have digital and physical completely separate folders.

You can do this already.  The Manage option lets you create any folder you want.

Hardware is integrated within the collection folder.  It isn't split into an isolated folder where you have to give special permissions to allow hardware to be in there.

I don't think it is reasonable or fair to restrict digital games to an isolated folder because people feel that their collection data is contaminated by said games existing.  I'm sorry if you feel that way but digital games aren't going away.

If splitting digital games into their own type is going to be similar to how hardware works then it's fine.  But to isolate everything and remove functionality (how I'm perceiving these feature requests to be) just sucks.


Sent from my iPhone using Tapatalk

I think you seeing separate folders/DBs as segregation/isolation/restriction/removal of functionality is the opposite of what would be happening.
Everything would finally get treated as it should be. Everything would have its own pages/search/etc tailored to it. Since hardware and then digital was added into a database made specifically with physical software in mind it has been like a square peg in a circle hole.

We have over 200 "consoles"(i use "" because digital delivery options are not consoles) listed in the database at this time because of the addition of digital delivery sources as a way to differentiate games from their retail counter parts.

Digital
1. As NES_Rules said we could cut a ton of wasted fields out of the fields on digital pages. Looking at a digital released game in the DB now it LOOKS like an empty page. Even when all applicable information is filled out.
1.a. I'm not a tech guy but i assume but cutting out all the waste fields(scans, data, boxes/manuals) would save resources. For example every single time someone does a missing image search it will hit a digital game page for 8 items when a digital platform won't have images for 4 of those fields. (Box front/back/ manual, game)

2. Having superfluous Box/Manual fields could be removed (quantity probably could too buy i know with Steam you can own multiple copies of the same game though i don't know why you wouldn't just want to trade/sell/give it away immediately.)
2.a. Again cutting the waste. We don't have to store empty values for an ever growing number of games in an ever growing number of collections. I assume this again would improve performance

3. Would increase ease of use. We still get Steam/GoG/Origin submissions for PC, PSN games submit for PSP/PS3/PS4, Games on Demand games submit for 360 with the variation title "Download" even though they do all have their own listing*
3.a. Saves time from staff having to reject a game and explain why. Also saves the frustration of members feeling like they wasted there time submitting this entry only to have it rejected.

4. I feel that in another years time the digital releases in the database will outnumber the physical releases. I also anticipate the creation of more and the support of more digital options. I really feel this would only further clutter search results and the system list and more than likely increase frustration of users.

Hardware
1. Much like the inclusion of digital "systems" clutters the list of actual systems the inclusion of Accessory, Cables, Controller, Memory/BackUp, System clutter the Genre field. When someone wants to search for a specific hardware item they shouldn't have to hunt through a list of completely unrelated things to find it.

2. Again ease of use. Items like cheat devices still get submit as Software. Wastes time of staff wastes time of user. No one feels good about having to reject a submission or getting a submission rejected.

3. Clutter/Ease of use. When you sort your collection (in real life) or go to buy from a store you don't look for the Playstation Move Controller between Pirates of the Caribbean and Power Gig.

Software (Physical) [Really doesn't relate at all but kind of needed to be here since i touched on the other 2... i guess.]
1. (PC specific)When digital started taking off and many people who heavily supported digital joined the site we ensured that they could view/track their collection in a way that worked for them by differentiating Steam/GoG/Origin/Etc from a PC retail release. Joyous cries were heard from the highest mountain to the lowest... low spot.

2. We added the virtual consoles/PSN/XBLA/GoD/XBLI for multiple reasons. People wanted to track that they owned I MAED A GAM3 W1TH Z0MB1ES!!!1 but didn't want it mixed in with their 360 games. While it is a game you play on the 360 Microsoft went through great lengths to let people know their system plays Retail/Games On Demand, Arcade and Indie games. Nintendo had done the same thing with the Virtual Console and WiiWare differentiating them in their own market places.

3. Variants have been(and probably always will be) a major point of contention. There are those completely against them. Those in favor of all of them and those in favor of some but not others. To a certain extent this problem has been dealt with for the users most irritated by variants returning when searching. When you want to know how many games were released for a specific system and learn that there are 8 versions of the same game bloating the number because of a potentially minor difference i can see how that would be frustrating.

*Initially all digital games fell under the umbrella of the system they worked on. Due to the ever growing demand that Steam and XBLA and PSN get their own support separate a split was made. That ended up opening up the doors for every other digital market place/distribution network for every other console.
When searching for an item to for any reason you wouldn't have to wade through a list of options completely unrelated to your search.

It is also worth noting we have never received complaints for not enough information being available in a search but do get complaints from too much making the user wade through to find the specific item(s) they want.



Really what it comes down to is a database split is in the best effort of:
Giving each item the right kind of attention
Keeping the database clean
Keeping everyone happy
Cataloging everything to the best of our ability. This is "The Classic and Modern Video Games Database" after all. Not the Cart-based only, CD-based only, Digital only, Hardware only, System/Company/Region specific Database.

Wall of text - TLDR, as I'm jumping back on Street Fighter. The advocacy of a "digital database" is due to how we currently separate software and hardware, and drive the pages based on that. Separating Digital would just be an extension of that - that's all that's being asked. As for the collection tool, it would just hide anything that was disabled.

I've never been a proponent of separating even more, as it would just lend itself to performance problems when querying multiple databases as mentioned earlier. Instead, we should be extending the software pages to handle digital as well.

If you can't be bothered to make a proper response why make one at all?

If you'd like a proper response...
Everything you're "complaining" about is due to limitations with our tools and page design, NOT the database. Dave, in his infinite wisdom, made a separate database to track hardware as he felt that it was the simplest thing to do to allow for a separate page template and completely expand upon it. However, he didn't do a proper split, it still has ties to the Software database, and it's causing problems. I'd rather NOT extend the problem and instead correct it at its source, but that leads into the extremely large refactor of the database that I've been poking at for months, trying to find a simpler solution rather than forcing our entire staff to deal with it. Extending the hack that is the hardware database with the digital database would work at first, but it will eventually cause performance problems as it grows.

If we wanted to do this now, it would only require a day or two's worth of work to get up and running in a rudimentary way. However, we would need to do something to address the performance problem in a year or two or risk significant problems later on down the line.

EDIT: For what its worth, proper "filters" and "shortcuts" would fix 75% of the complaints that I've heard from people on the site.

I think the source of all the problems has and will continue to be the forced integration of "databases" that should be fully separated.

It really seems that saying there is a hardware and a software database is like saying there is a 360 and a PS3 database. The only major difference is a little ID in the RFG ID system.

Even if i select Hardware on the search menu it isn't like it removes all non-hardware related items from the other search options.
No mater how many times i search for it i am never going to find a piece of hardware that is in the Action/Adventure genre.

That's what I'm getting at about the broken filters - "Genre", in the backend, is referred to as "Type" in both databases. Splitting the databases may help for what you see visually, but it's something that can (and should) be fixed without causing other problems. The usability problems you're encountering are usability problems that aren't related to a database split.

For now, considering that we've seen how quickly I've jumped on this after we discussed it a year or two ago, I'll create the separate pages and make the appropriate changes, though at this point I may as well make the Strategy Guide and Arcade databases if I'm going through the effort...

after reading through all the long posts, i think i'm leaning more towards what i believe Shadow is saying. which is (i think) to keep everything in one database, but to have a way of visually separating them to users. so on the backend, nothing has really changed, but on the front end it'll look completely separate.

i see it possibly working on a way of:
1. go to database search
2. it asks you whether you want to search for hardware, physical software, or digital software.
3. it visually hides all fields unrelated to what your choice above was, so that to the user it looks like a completely different search page/database, even though on the back end it still processes things through the same place.

the only thing i don't quite understand still is why splitting the database is worse for performance, if your search filters say "search hardware only", wouldn't that mean the servers have to look through much less data to find what you want?

^That is what i have been trying to get at all this time.

I just can't understand the argument for having the ever increasing cluttered results and search functions if they are already 2 separate DBs they should be treated as such.
And luckily adding digital should be less of a hassle since we have done a good job of keeping all the info from being mixed in with physical releases already.


Sounds good! Maybe instead of being just strat guides though it could include more mags like Nintendo Powers, Tips & Tricks, Sega Visions, etc, etc
Maybe even some other printed materials?

The problem is if you want to search everything.  I want to personally be able to search both digital and physical at the same time if I'm searching for something to see what platforms/distributions it's on.  It's less useful if I have to do two separate searches.

I don't think mysql supports cross database joins and queries, so if I wanted to query digital games in database D and physical games in database P in one fell shot (this is on development level, not on user level), then you have to run two separate queries, get the results from both back into memory, and then sort them in memory instead of using the mysql sort logic which is going to be multitudes faster on large data sets. 

ahh see that's the explanation i was missing, that makes sense.

@izret sorry i thought you were in the "split the databases in two" camp, it's all very confusing :P i think i understand more of what everyone is saying now though.

I am :P
Added a couple extra words to help clarify.

While the argument of wanting to search to see everything does have validity it IMO is what the tool is least used for and will continue to be least used for in the future.

On the rare instances someone is going to be searching to see how many platforms / distribution services i think having to search in each of the databases for those specific instances would be less of a hassle than having to comb through all the "systems" / "genres" / unrelated results on a daily basis.

Personal example:
As someone who is only buying games from a specific series at the moment i use the database to search for just Mortal Kombat. I do i get the benefit of seeing every system/region it has been released for. In addition to the PSN, Steam, etc releases. While unlike some others i DO support digital I don't want those results returning in my searches.

I am not going to jump on eBay and find digital copies of those games, i'm not going to talk to some guy in Europe and see if he can find a copy of MK Komplete for Steam at GAME.
So i keep scrolling past them and have to readjust my count every time a new download is added.

We have multiple users with "junk" folders so they can stop things coming up in their searches. All that does is skew the site statistics. Users having to create "junk" folders to force better search results should be really telling that there is a problem with how things work now.

...just to clarify what you expect:
Current Behavior:
Search Software, get Software results.
Search all, get all results.

Expected Behavior:
Search Software, get Software results (minus Downloadable Media Format)
Search Digital, get Software results with ONLY Downloadable Media Format
Search all, get all results.

Is this correct?

I think what you're talking about here is a lack of filters for what you want to search for.  There's no exclude functionality for types.  When you search for stuff, types is a drop down.  A list of checkboxes would probably solve that problem.  I think the real problem is people are coupling the database architecture with the layout of the UI.

Honestly, splitting the database is not necessary.  Multiple tables in a single database, yes - that would be better, but using two databases when tables would suffice - that's just major overkill.  

Zthun, he's referring to tables, not databases. We currently have a table for games and a table for hardware under rfgene2_games.

Are the db schemas published anywhere for us to peruse?

Ah, ok, that's fine then.  So my whole argument is moot at this point.  When you guys were talking about splitting the databases, I thought you meant you were going to split two databases where you would have to do a switch on the db connection in order to query it.  

Then my original point of performance is no longer valid.  You can just the same query multiple tables in the same database.

Normally, when I hear database, I think of the legitimate database object near the root of the server.  Basically, I thought you were talking about doing something like (note, this is for mysql only):

[Server]
    [SoftwarePhysicalDatabase]
            [TableA]
            [TableB]
            [...]
    [SoftwareDigitalDatabase]
            [TableA]
            [TableB]
    [HardwareDatabase]
            [TableA]
            [TableB]

I was like...that's just overkill when you can just do this:

[Server]
    [GamesDatabase]
         [TableCommonData]
         [TableSoftwarePhysical]
         [TableSoftwareDigital]
         [TableHardware]
     

Surprised so many people are actually interested in the site layout... To be honest, I could always use a hand, especially on the UI side, in getting the site up to modern standards. The PHP 5.5 upgrade (from 4.0) broke quite a few things that we STILL haven't fixed.

In terms of how it's laid out, Zthun's example is pretty close:
[rfgene2]
    [rfgene2_forum]
        [....]
    [rfgene2_games]
        [...]
        [games]
        [hardware]
        [...]

Worth noting that the DB schema, and many of the columns, are in sore need of a refactor. Keys aren't properly used in many instances, which causes a lot of problems - especially with strings.

for me i just like hearing technical explanations of why things do or don't work, it's much nicer than a generic "sorry we can't do this" or "that's a bad idea" with no real explanation. transparency is awesome :)


i wouldn't mind helping out, but i might be a bit slow at first because i would have to do some research to learn more about php. i could definitely help with hunting down/fixing minor UI bugs, typos, etc around the site.

Happy to advise or potentially code. Do you guys have the schema and code up anywhere to peruse ?

No - we're not really equipped to actually collaborate for code right now, especially with the fact that we don't even have individual accounts to track or limit access to our database and server. Now that we're getting more people that have more than a passing interest in helping out though, I've definitely been considering it.

this site might help too with tracking bugs and showing who's working on what: https://trello.com/

Funny, trello was being used by one of our managers today. Actually, we're using Mantis here on the site.
https://www.mantisbt.org/