KatsBits Community

General Category => Public Discussion => Topic started by: silicone_milk on February 09, 2010, 07:20:03 PM

Title: Where I've Been
Post by: silicone_milk on February 09, 2010, 07:20:03 PM
I kind of disappeared after setting up some projects for myself (q3map2 executing on the GPU and my website).

The reason for this is I landed a full time job. I currently work at an IT consulting company. I handle enterprise level database administration, data analysis software development (think google analytics for MS SQL, MySQL, and Oracle databases), network security, and in-house software development (auto status reports on all active servers, ez-sync programs to get computers with fresh OS installs updated with necessary updates and necessary software before being connected to the internet, etc...)

So, yeah. I've been really busy. I also just moved in to a new place three weeks ago and I still haven't gotten around to hooking my desktop back up :P

Looking forward to pay-day so I can get my hands on a cintique 21ux (http://www.wacom.com/Cintiq/cintiq-21ux.php) :)
Title: Re: Where I've Been
Post by: kat on February 10, 2010, 05:06:44 AM
That's a bit of a departure isn't it? Going from a game developer into IT? Are you doing any game related stuff at all now?
Title: Re: Where I've Been
Post by: silicone_milk on February 10, 2010, 05:22:48 PM
Hm.. I suppose it is.  ;D

I'm still doing game related stuff... just not as much nor as often.

I have a couple contracts to finish up at Chronic Logic and I'm not sure what's going to happen after that. I still dream of working for SquareEnix or Kojima Productions some day :)
Title: Re: Where I've Been
Post by: ratty redemption on March 05, 2010, 07:52:46 PM
this is another topic I previously missed *ratty bangs head on desk*

@ milk, good luck with your new job. my gf works in IT for a london insurance company heading up their help desk among other things. she's found it very stressful at times, I hope your job doesn't turn out like that.
Title: Re: Where I've Been
Post by: silicone_milk on March 05, 2010, 07:57:13 PM
haha no worries. It's not too stressfull. I just work with people from Peru and Mexico with really thick accents so communication breaks down at times.

They also don't listen to my suggestions. Such as the security measure that just requires us to connect to a machine at our shop then to our clients computers to encrypt our traffic to protect sensetive information. According to them it's "too complicated"
Title: Re: Where I've Been
Post by: ratty redemption on March 05, 2010, 08:30:25 PM
@ milk, understood and one of my gf's suggestions of not leaving laptops containing clients sensitive data on trains was also dismissed for similar reasons.

considering how incompetent her bosses are, it's amazing they make any money, but they must be doing something right has they keep buying out smaller companies. to which my gf has tried to explain to them that they should spend some of their profits on employing new IT staff but apparently they don't understand the concept.
Title: Re: Where I've Been
Post by: silicone_milk on March 16, 2010, 06:44:07 PM
bought a gts 250 and picked up work on q3map3 (q3map version 3.0.01)

CUDA, for the record, is a royal pain in the ass to set up a proper development environment for (in windows at least).
Title: Re: Where I've Been
Post by: ratty redemption on March 16, 2010, 07:10:44 PM
@ milk, how does nvidia's cuda compares to ati's equivalent if there is one?
Title: Re: Where I've Been
Post by: silicone_milk on March 17, 2010, 12:59:38 AM
not sure. Haven't touched an ATI card in many many years.
Title: Re: Where I've Been
Post by: ratty redemption on March 17, 2010, 06:11:37 PM
@ milk, understood.

do any of you other guys know?
Title: Re: Where I've Been
Post by: Shaderman on March 17, 2010, 08:10:34 PM
ATI (AMD) has the Stream technology (http://www.amd.com/US/PRODUCTS/TECHNOLOGIES/STREAM-TECHNOLOGY/Pages/stream-technology.aspx).

Here's something like a comparison. (http://www.tomshardware.com/reviews/amd-stream-gpgpu,2335.html)
Title: Re: Where I've Been
Post by: ratty redemption on March 18, 2010, 05:55:41 PM
@ shaderman, cool and thanks for the links.
Title: Re: Where I've Been
Post by: silicone_milk on March 18, 2010, 06:53:47 PM
agreed. Was an interesting read.
Title: Re: Where I've Been
Post by: silicone_milk on March 18, 2010, 11:26:06 PM
As I've been doing some more research I've been more and more aware of the fact that a lot of people have ATI cards rather than NVIDIA.

Unfortunately, CUDA is an NVIDIA exclusive technology and will only run on their cards. This cripples the ability to distribute q3map3 and ostracizes non-NVIDIA users.

I've shifted focus from CUDA to the younger and more experimental OpenCL which will allow the program to make use of both the gpu and cpu to balance workload (instead of gpu only as in CUDA) and is device agnostic. This means that q3map3 will be able to run on any device with opengl support. (Mac, Linux, and Windows users with either ATI or NVIDIA cards will be able to run it).

That is all.
Title: Re: Where I've Been
Post by: silicone_milk on March 19, 2010, 06:04:37 AM
So, a couple updates:

OpenCL, being a fairly new tech, doesn't seem to want to work properly on my machine even though I have an OpenCL compatible video card. Whatever.

Secondly, this is going to take much longer than I thought as the GTKRadiant and Q3Map2 sources are heavily coupled (they depend a crapload on each other). I'm in the middle of decoupling the code. I'm thinking about keeping the q3map2 code interface the same so existing GTKRadiant sources can build with the q3map3 source, but this is only benefitial to people who want to compile the Radiant source with the new compiler.

Title: Re: Where I've Been
Post by: Shaderman on March 19, 2010, 06:50:54 AM
Secondly, this is going to take much longer than I thought as the GTKRadiant and Q3Map2 sources are heavily coupled (they depend a crapload on each other).

Oh yessss, I remember that from GtkRadiant 1.5. I don't know which version you're using (I guess 1.5 or 1.6), but maybe it's worth to look into the other one to get out the q3map2 source. AFAIK q3map2 didn't change from 1.5 to 1.6 but since the 1.5 source was heavily rewritten in all parts it's maybe more coupled with q3map2 and thus it might be much easier to get q3map2 from the 1.6 code base which itself is based on GtkRadiant 1.4.

It's 7:50 am here and I didn't have a coffee yet... I hope this makes sense :)
Title: Re: Where I've Been
Post by: silicone_milk on March 19, 2010, 07:50:48 AM
it does make sense. I'll take a look in to it. I've been working out of the 1.5 codebase and trying to make the q3map2 a completely independent project (as part of the main focus on running it on the gpu). Also, since I'm at it, I'm rewriting the base in c++ to have access to the boost library and the standard c++ libraries. Although, I'll have to work some c in to the parts that will be running on the gpu since the c++ extensions aren't fully developed yet...
Title: Re: Where I've Been
Post by: silicone_milk on March 20, 2010, 11:21:13 PM
Got the BSP section of the q3map2 source compiled on its own. Replaced all the raw pointers with boost smart pointers and all the c-strings with standard library string types.

Introduced some interface classes to work on BSP data rather than having a bunch of loose global functions floating around in the q3map2 header.

Having some issues with the fact that q3map2 relies heavily on almost all of the variables being global. There's really going to be a lot of work refactoring all of that code.
Title: Re: Where I've Been
Post by: Shaderman on March 21, 2010, 06:14:11 PM
That's a start :) Oh well, I remember those static vars everywhere in the Radiant code  ::)
Title: Re: Where I've Been
Post by: silicone_milk on April 01, 2010, 05:03:48 PM
Got some more work done on it. Embedded the Python interpreter in to the compiler.

More detail can be found at the Quake3World thread - http://www.quake3world.com/forum/viewtopic.php?f=10&t=43211 This is where I've been talking about it mostly.
Title: Re: Where I've Been
Post by: Shaderman on April 01, 2010, 10:50:56 PM
Nice! I'm very interested in this technology and I'd love to see the code changes you made to q3map2 once you've something to show.
Title: Re: Where I've Been
Post by: silicone_milk on August 17, 2010, 11:37:54 AM
nothin to show.

Adding some test code to the heavily coupled q3map2 source from the gtkradiant 1.5 svn repo to see if I can offload some basic functions to the gpu first.

The C++ route was just too much damn work for a hobby project. And the opencl bindings are for C anyways so might as well just stick with it, neh?

I *did* set up ssh and a git repo (svn but better) on my server and once I lock down the ssh access with chroot to keep people from having full access to my server, I'll be opening the repo as read-only to people interested in keeping track.
Title: Re: Where I've Been
Post by: carnage on August 18, 2010, 01:23:24 PM
Interesting project Milk. Having done a fair bit of GPU programming myself it will be interesting to see what kind of performance benefits you can get from this. Its defiantly not a trivial matter to move a CPU code into a massively parallel architecture with very limited inter thread communications.

I haven't worked with CUDA or CL though. Will be interesting to see if this is even possible.
Title: Re: Where I've Been
Post by: silicone_milk on August 19, 2010, 02:39:01 AM
Thanks. I can see a huge performance increase (theoretically) for the light compiling phase as we're dealing with a lot of floating point math in huge matrices.

OpenCL is a joy to work with. Makes getting code on to the GPU quite easy and has a similar syntax to the ANSI C99 standard of the C programming language.

I'm going to chroot my server tonight I guess. And get that git repo public :)

At the moment I'm working on a kernel for radiosity calculations.
Title: Re: Where I've Been
Post by: ratty redemption on August 19, 2010, 03:29:38 AM
milk, i can't remember but did ydnar ever get ao to work with q3map?
Title: Re: Where I've Been
Post by: carnage on August 19, 2010, 09:52:01 AM
Interesting, I have only worked with sharer level graphics programming so I guess that general purpose GPU programming languages are a bit more lenient with amount of variables/memory you can access to be able to do some cool stuff like this.

I would love to see the source when you get the server up so I can really see what going on under the hood and get a better idea of what GPGPU stuff is capable of.

But in general graphics cards are pretty good for processes that can be split up into a large number of independent processes that have roughly the same execution time (AFAIK). So I guess quite a bit of q3map is a good candidate for acceleration.
Title: Re: Where I've Been
Post by: pazur on August 19, 2010, 10:28:54 AM
Would be great if you can make this GPU q3map for ATI users although I don't map on the id-Tech 3 engine anymore I might consider building a custom map for Urban Terror in future.
Title: Re: Where I've Been
Post by: silicone_milk on August 19, 2010, 10:48:29 AM
Would be great if you can make this GPU q3map for ATI users although I don't map on the id-Tech 3 engine anymore I might consider building a custom map for Urban Terror in future.

OpenCL is able to execute on ATI cards although I have none at my disposal to test the code on. The beauty of OpenCL is that it's a standard that is implemented to run on many platforms. The biggest community for OCL development I've seen happens to be made up of mainly Mac programmers surprisingly enough.

CUDA, on the other hand, is an NVIDIA only tech that I feel will probably end up being phased out since NVIDIA is supporting OCL now as well.

Didn't get around to fixing my server tonight. They kept me late at work and I'm a bit tired.
Title: Re: Where I've Been
Post by: silicone_milk on August 26, 2010, 11:31:08 AM
Hm. Forgot I said I was going to open the git repo.

Spent the past three days fighting a buffer overflow bug that only appeared in the release version of q3map2 but not in the debug o_O

It was a silly little fix that worked effectively. Took forever to track it down.

So there *is* something I haven't tested yet regarding read/write permissions to the repo. I don't think you guys would ever upload anything retarded to my repository but I like to minimize possible... issues.
Title: Re: Where I've Been
Post by: kat on August 26, 2010, 04:22:11 PM
Don't open a server to the Net, if that's what you're talking about, it's just asking for trouble if you have no real way to monitor what's being uploaded to it.
Title: Re: Where I've Been
Post by: silicone_milk on August 30, 2010, 08:15:19 AM
I locked down an account to run with a restricted shell and have limited access to tools but I never thoroughly tested it to make sure everything's water-tight.

Right now Misantropia over on Quake3World holds a ssh key to access my source code. The plan is to just upload it to github for public access.

All current updates can be found on my thread over on quake3world here: http://www.quake3world.com/forum/viewtopic.php?f=10&t=43211

More work has been done. Getting closer to getting the first kernel running ;)
Title: Re: Where I've Been
Post by: silicone_milk on September 09, 2010, 01:07:48 AM
First kernel's running. See q3w thread for examples.

Seeing some good performance increases on non-optimized code :)

Busy with school right now but I'll be working on refactoring the code so the kernel isn't hardcoded and we'll have a more flexible solution to play with.
Title: Re: Where I've Been
Post by: silicone_milk on February 09, 2011, 12:45:50 PM
Still workin on q3map3.

Updates are to be found on the quake3world thread previously linked here.

Am running with a startup indie game developer now over in San Jose, CA. We're aiming for a game released on the xbox live arcade market by the end of the year.