Post new Lua scripts here!

[Wyst3r]

Introduction:

Back in February, Grav asked me if i had any memory watchlists or Lua scripts to share. Since I've been doing memory watching for a long time, I had plenty of useful addresses, but for whatever reason, no Lua scripts. I realized at that point just how useful such scripts can be and how much of the manual work I tend to do that could be automated. The programmer in me immediately started thinking up wild, crazy ideas for scripts, and one of the first things that came to my mind was a real-time map. One where you could see the location of Bond and every guard in the level,  as well as their current state.

I knew enough about the games memory to pull that off, but I had never written a single line of Lua code before. Thankfully, it turned out to be a very simple language (I normally code in C++) to learn and it quickly became one of my favorite programming languages. It didn't take long before i had a working prototype ready. The first version took the shape of a minimap and was extremely simplistic. I realized that in order to make something that would actually be useful, I needed something more than just relative guard locations, I needed an actual map of the level.

The Goldeneye Setup Editor was a huge help at this point, since I would never have been able to find and extract the clipping data by myself. The export clipping to .obj feature was very convenient, and with a bit of pre-processing i could finally see a full blown pixel-perfect map of the levels! There was only one problem though... Performance. The map was ridiculously slow, as can be seen in this early sneak peek: https://www.youtube.com/watch?v=E80LCF4RHWw I knew i had to optimize it somehow, and I had a pretty good idea about how to do it: Quadtrees. I didn't know if this would be enough, but after a bit of implementing of testing it became clear that Quadtrees was a lifesaver for the project. Drawing lots of lines on the screen was still fairly slow, especially on levels like Frigate, but still far better than the old version.

As the project went on, I kept coming up with new ideas that i wanted to implement. The prototype project turned into a full blown implementation with lots of files and thousands of lines of code. In May, I began to see the end of the project, and made a short list of stuff I wanted to fix before the first release. And last week, I finally reached the finish line...

Trailer:
Not a valid vimeo URL
How to run:

To run the script, follow these steps:

1. Get the latest version of BizHawk (tasvideos.org).
2. Get a Goldeneye rom file.
3. Clone the git repository OR download the zip file:

Git: https://bitbucket.org/Wyst3r/bizhawklua
Zip: https://www.dropbox.com/s/mpg03gscqmma61c/BizHawkLua.zip?dl=0

4. Load Goldeneye in BizHawk, and go to Tools -> Lua console.
5. Go to Script -> Open Script, and open Map.lua from the root folder.
6. That's it!

Controls:

To pan around the map, hold the left mouse button.
To select a target, click the left mouse button.
To switch camera mode, press M.
To switch floors, press F.

Known issues:

There a few things I know about but either haven't gotten around to fixing yet or simply don't know how to fix. I'll make a list later. If you find any bugs/issues or have suggestions, feel free to post them here.

Future:

There's still plenty of stuff that could be added to this (I have a long TODO list on my hard drive). I don't know yet if I'll continue working on it now or take a break (development has been quite intense these last few months, so probably the latter). I also have plenty of ideas for other scripts that would be interesting to create.

Reusability:

The classes in the Data folder are very generic and could be reused for many other applications. If you wish to write your own Lua script for Goldeneye, they're a great place to start. Over time, my hope is that the gaps in the data can be filled in and we can have a somewhat complete toolkit for easily accessing Goldeneye data.

Bonus:

Some funny stuff that happened randomly while messing around with memory addresses.

Other scripts:
File Recorder - https://forums.the-elite.net/index.php?topic=21141.msg437012#msg437012
Lag-O-Meter - https://forums.the-elite.net/index.php?topic=21141.msg450439#msg450439

[Grav]

Excellent work! 

For when the thread is renamed, here is my very basic File Recording LUA Script that should be easy for beginners to use. I used this primarily to test how speed works in this game (I would put the data into a spreadsheet and make graphs), but this can be applied to anything you want to test.

Here is an example of graphs I generated from this script:

Spoiler

Code: [Select]

while true do

--[[
Bond memory domain for each level
0x0B3CA8  - Bunker 1, Silo, Bunker 2
0x0C54A8  - Statue
0x0C04A8  - Control, Train, Jungle
0x0CCCA8  - Archives, Egypt, Caverns, Cradle
0x0C68A8  - Frigate
0x0B00A8  - Aztec
0x0D6CA8  - Streets
0x0D94A8  - Depot, Runway, Surface 1
0x0D30A8  - Dam
0x0B64A8  - Facility
0x0E5CA8  - Surface 2

Hex[Base 10] offset from level domain start
+00[0]     - Angle (Horizontal)
+10[16]    - Angle (Vertical)
+344[836]  - Position X (north south)
+348[840]  - Position Y (up down)
+34C[844]  - Position Z (east west)
+3D8[984]  - Speed X
+3DC[988]  - Speed Y
+3E0[992]  - Speed Z
+938[2360] - Noise

**You must use the base 10 offset when adding to the domain start in this lua script**
]]


--XZ Planar Speed
xzs = math.sqrt((mainmemory.readfloat(0x0E5CA8+984,true))^2+(mainmemory.readfloat(0x0E5CA8+992,true))^2)

--Outputs values per frame to file in bizhawk lua directory. Replace filename/type and written variable with w/e.
local file = io.open("speed.txt", "a")
file:write(xzs, "\n")
file:close()

emu.frameadvance()
end
Save this code as a .lua file and use it with Bizhawk just as you would with Henrik's script. Alternatively, you can download this directly here:
https://drive.google.com/file/d/0BxzIQ78p3ZfDQzlGWTIybkZVNE0/view?usp=sharing

If you want to edit these files in any way, I recommend using something like Notepad++ or Sublime. Also, feel free to post any additional useful memory addresses related to bond or whatever that can be added to this script so people don't have to go searching for it again.

[Wyst3r]

That is a great example of a script that can be simplified with the new data classes btw. To get the player speed:

Code: [Select]

require "Data\\PlayerData"

PlayerData.get_value("speed")
Also you might have noticed I render Bond's speed vector in the map. Might give some extra insight about how it works.

I've noticed that the speed vector is the biggest factor in determining if warps will succeed or not. If it intersects any collision volume, the warp will always fail. However, if the vector is aligned perfectly to avoid collisions, you almost always make it. Since the vector wobbles back and forth all the time it's a bit hard to align it perfectly, but knowing roughly where to aim should still be useful. Especially for stuff like the Depot warp.

[Grav]

Yeah anything that can simplify stuff would be nice. I still don't know hardly anything about all of the advanced stuff so I will leave improvement/optimization to those who do.

Speed specifically is just an example of what I recorded... I'm sure theres a lot of other stuff we can learn more about.

[Grav]

Here are a couple error messages I get. The map seems to work though (though my bizhawk runs at 15 fps with it on)

"Error message: A .NET exception occurred in user-code error running function attached by lua function event.onframeend"

"Error message: [string "main"]:998: attempt to call field 'SetGameExtraPadding' (a nil value)
error running function attached by lua function event.onframeend"

[Wouter Jansen]

Kreygasm

[Wyst3r]

Did they remove SetGameExtraPadding in latest BizHawk? Ive only tested this with 1.11.4 and assumed it would still work in 1.11.6. But maybe not...

Also getting console output while running will destroy your fps. Try 1.11.4 instead. (30ish fps shpuld be worst case scenario, on Frigate. Levels like Runway should be a stable 60 fps)

[Grav]

I was using 1.11.1. Also yeah I figured the console output could be why

Also you should update TASvideos forum with this lua script. This stuff can be applied not only to GE but to tons of other games probably. I know a blast corps TAS would be able to make very good use of this map overlay.

[Taylor]

Wow, I really can't overstate how cool this is. You are seriously doing great work here mate!

[007]

Very impressive!

[Wyst3r]

Lag-O-Meter

Based on my recent Facts topic post on lag, I made this little script that displays the current amount of lag. The meter itself is based on accurate decimal numbers, but since the game always rounds to the nearest integer, I also highlight that integer in purple. Getting below these integer thresholds is the key to saving time.

Download:
https://www.dropbox.com/s/5nd9eh5jvpthzd0/Lag-O-Meter.lua?dl=0

[Whiteted]

Henrik that GE map is incredibly useful and positively inspiring. I'm new to lua and I've had trouble trying to work out how to enumerate the objects: For instance if I wanted to get the positions of all cameras in the level how would I go about doing this?

On topic, I've also found this little script for randomising the RNG when a state is loaded to be very useful for human-testing and generally practising strats:

Code: [Select]

-- Randomises the RNG's state when a savestate is loaded, unless the ` key is held.

-- The address containing the RNG's state: read from Wyster's GE map
rngAddr = 0x024464

-- Randomise our seed using the current time, and cook it in.
math.randomseed(os.time())
for i = 1,64 do
    math.random()
end

-- Writes the random 4 bytes to the address and to the log
funcName = "randomiseRNGstate"
function randomiseRNGstate()
    -- Only randomise if the Grave (`) is not held.
    if input.get()["Grave"] ~= nil then
        console.log("State loaded without randomisation.")
    else
        -- math.random doesn't seem to like large numbers..
        local vb = math.random(0,65535)
        local vl = math.random(0,65535)
        local v = vb * 65536 + vl
        mainmemory.write_u32_be(0x024464,v)
        console.log("State randomised to " .. v .. ". Hold ` on load to prevent")
    end
end

-- Unregister any existing functions of the same name
while event.unregisterbyname(funcName)
do
    console.log("Existing registered 'randomiseRNGstate' removed")
end

-- Register the function
event.onloadstate(randomiseRNGstate, funcName)

Complete with comments, as standard 

[Wyst3r]

I think all you have to do is this (haven't actually tested it):

Code: [Select]

require "Utilities\\GE\\ObjectDataReader"

ObjectDataReader.for_each(function(object_data_reader)
        if (object_data_reader.current_data.type == 0x06) then
                local position = object_data_reader:get_value("position")
        print(string.format("{%f, %f, %f}", position.x, position.y, position.z))
        end
end)

[Whiteted]

HUD

I'm home for the summer and the past couple of days I've been working on a Head-up display for emulating Goldeneye. The idea is to be able to mark points in the 3D world which are not visible for whatever reason (i.e. they are behind a wall). I've written the main (and very small) core which you can build whatever you like on. The main thing I'll try to do is integrate it with all the work that Henrik has done in the GE map to provide a form that you can select which objects you want to mark from (and have nice colours and the like).

The main achievements of the core so far are:

• Managing to project 3D points on the screen seemingly without bugs!
• Reads azimuth and inclination angle values which appear to account for Bond's aim wobbling
• Copes with weapon-zoom, 16:9, wide/cinema/the other one [\li]
  

Currently it just marks all guards in red.

The main issue with the core is that the HUD appears to read updated positions and bond-angles when the game has not drawn another frame. Does anyone know the address to read for if it is a lag frame? Particularly Henrik since I assume this required it:

I modified a ROM to force the number of frames passed to a constant value, essentially emulating constant lag.

The code (include the Data folder from Henrik's GE map https://bitbucket.org/Wyst3r/bizhawklua)

Code: [Select]

-- Needs access to Wyster's data from the GE map directory.
require "Data\\GE\\PlayerData"
require "Utilities\\GE\\GuardDataReader"

-- ============== HUD v1.732 ================
-- Currently draws red squares over all alive guards who are in the field of view.
-- Could easily be adapted to mark objects, i.e. the radios in surface 2.

-- ========= Improvements on v1.0 ===========
-- Updates only when frames are drawn
-- Caches camera early: no lag even as Bond turns sharply, nor on zoom
-- Bond wobble accounted for: The HUD properly accounts for Bond's aim swaying slightly
-- Look-down / up fixed: HUD no longer gets severely warped

-- ============= Minor issues ===============
-- Occassionally a draw happens 1 frame earlier than we predicted, and so HUD is behind.
--   This mistake does not snowball, and is probably unnoticable when playing in real time.

function dotProduct(v,w)
    return v.x*w.x + v.y*w.y + v.z*w.z
end
function vectorSubtract(v,w)
    return {x=v.x-w.x, y=v.y-w.y, z=v.z-w.z}
end
function scaleVector(v,a)
    return {x=v.x*a, y=v.y*a, z=v.z*a}
end

-- Determines the size of the border due to wide,cinema etc.
function getBorder()
    local border = {}
    border.topBase = mainmemory.readfloat(0x07C114,true)
    border.bottomCeil = mainmemory.readfloat(0x07C11C,true)
    return border
end

-- Determines all important properties of the camera in this instance,
--   assuming that the camera is "Bond's eye" (which is not the case in most cutscenes)
function determineCamera()
    local camera = {}

    -- "Zoom" : treated as the distance from camera eye to the viewing rectangle.
    --      This is equivalent to just scaling points 'outwards' by this factor
    --      Results suggest we are interpretting it correctly.
    camera.zoom = mainmemory.readfloat(0x0607E0,true)

    -- Get camera location - assumed to be at bond
    camera.loc = PlayerData.get_value("position")

    pdStart = PlayerData.get_start_address()

    -- We are satisfied that these represent the wobbled-inclination sine and cosine
    sinI = mainmemory.readfloat(pdStart+0x04C4,true)
    cosI = mainmemory.readfloat(pdStart+0x04D0,true)
   
    -- But other nearby values seem to be slightly conflicting products which we can't deal with.
    -- So for the azimuth values, we use the 'left_wobble' that we found,
    --   we mimican Observed relation between 'up_wobble' and the sinI,cosI values above.
    -- This is adding 1/sqrt(3) * the left_wobble to the azimuth angle.
    up_wobble_addr = pdStart + 0x0530
    left_wobble_addr = pdStart + 0x052c
    left_wobble = mainmemory.readfloat(left_wobble_addr,true)
    A = PlayerData.get_value("azimuth_angle")
    adjusted_A = A - left_wobble/math.sqrt(3)

    -- Convert to radians and determine the sine, cosine.
    adjusted_A = adjusted_A * (math.pi / 180)
    cosA = math.cos(adjusted_A)
    sinA = math.sin(adjusted_A)

    -- 0 degrees azimuth is +z, 90 is -x.

    -- The camera's unit direction. Relatively straightforward
    camera.dirc = {["x"]= -sinA*cosI, ["z"] = cosA*cosI, ["y"] = sinI}
    -- The camera up vector. This has the same azimuth but with inclination increased 90 degrees.
    --   This relies on broadening the idea of inclination angle, which is normally in the range 270-90
    -- Recalling cos(x) = sin(x+90), -sin(x) = cos(x+90), we have
    camera.up = {["x"]=-sinA*(-sinI), ["z"]=cosA*(-sinI), ["y"]=cosI}
    -- With some thought, observe that the cameraRight vector has y = 0 (camera has no roll)
    -- Then the azimuth is incremented by 90. So as above sinA->cosA, cosA->-sinA, cosI->1:
    camera.right = {["x"] = -cosA, ["z"]=-sinA, ["y"]=0}

    return camera
end

-- Takes a point p and performs our projection to get the vector in the virtual screen,
--  then maps it onto the actual screen. Returns nil if it isn't in view.
-- For resolutions different to 320x240, say AxB, change K to be A/2 and this should work.
-- The center point in drawHUD will also need to move.
function getHUDpoint(p,camera,border)
    -- Get the displacement from the camera, and ahead component length.
    v = vectorSubtract(p,camera.loc)
    dp = dotProduct(v,camera.dirc)

    -- Insist that the point lies in front of the screen, else drop it
    -- Potentially this should be 1/zoom..
    if dp >= 1 then
        -- Get the vector within the virtual screen
        screenV = vectorSubtract(scaleVector(v,1/dp),camera.dirc)
       
        -- Now project onto up/right vectors
        up = dotProduct(screenV,camera.up)
        right = dotProduct(screenV,camera.right)

        -- Scale in using zoom: literally multiply by it
        -- ALSO by my magical K. Without K it appears that the edge of the screen is 1 unit from the center.
        -- K is half the width of the screen. Oddly this seems to also be correct value for adjusting height too.
        K = 160
        up = up * camera.zoom * K
        right = right * camera.zoom * K

        -- Check the point is in the viewing frustrum (accounting for the borders too)
        if right >= -K and right <= K then
            if up + 0.75*K >= border.topBase and up + 0.75*K <= border.bottomCeil then
                -- If so return the point to draw
                return {x=centerP.x+right, y=centerP.y-up}
            end
        end
    end
    -- Return nil for no point
    return nil
end

function getHUDps(camera,border)
    nHUDps = {}

    function projectGuard(g)
        -- Presumably could use get_value equivelently
        pos = g:get_position()

        p = getHUDpoint(pos,camera,border)
        if p ~= nil then
            table.insert(nHUDps,p)
        end
    end

    -- Call the guardDataReader to populate nHUDps
    GuardDataReader.for_each(projectGuard)

    return nHUDps
end

function drawHUD(ps)
    -- Draw the center point green (potentially a pixel off)
    centerP = {x=160,y=120}
    gui.drawRectangle(centerP.x-1,centerP.y-1,3,3,0xFF00FF00)

    -- Draw all other points red atm
    for _, p in pairs(ps) do
        gui.drawRectangle(p.x-1,p.y-1,3,3,0xFFFF0000)
    end
end

-- =====================================================

local counter = mainmemory.read_u32_be(0x04837C)
local updateOn = {}
local HUDps = {}

-- Loop allows for toggling of the script.
while true do
    -- Odd hack to find out if we are drawing a frame.
    -- Seems that this global timer or counter increases 2 frames before any draw.
    prevCounter = counter
    counter = mainmemory.read_u32_be(0x04837C)

    fc = emu.framecount()

    if prevCounter ~= counter then
        -- A draw has begun. Store the previous frame's camera, ready for when the frame has been drawn
        updateOn[fc+2] = prevFrameCamera
    end

    -- If we predicted a draw would finish on this frame, remember the correct camera, and update the HUD
    if updateOn[fc] ~= nil then
        -- Get fresh border and use current positions.. but the previous game-frame's previous frame's camera.
        border = getBorder()
        HUDps = getHUDps(updateOn[fc], border)
    end
    -- Destroy the old camera: no memory trouble here.
    updateOn[fc] = nil

    -- Draw the HUD again regardless (potentially with the same image)
    drawHUD(HUDps)

    -- Store the current frame's camera, for if we detect that a draw has begun in the next frame.
    prevFrameCamera = determineCamera()

    emu.frameadvance()
end

Also available at https://github.com/whiteted-strats/GE_HUD/blob/master/HUD_root_3.lua

Edit: correcting code to use playerData
Edit: updated to new version: much improved (see github)

[Wyst3r]

Cool! I've been wanting to do something that for a long time, so really nice to see it done. I always figured you'd need to find the internal projection matrices or something to do this, but I guess some basic linear algebra was enough

Some bugs/issues I noticed:

1. The addresses used for inclination/azimuth are static, which means they will only work on certain levels like Runway. You need to use the PlayerData class to acquire those values if you want them to be dynamic and work on all levels.
2. It's hard-coded for 320x240 resolution. Personally I usually use something like 640x480 so it's a bit annoying Should be quite easy to fix though, you can check my map script to see how I handle it.
3. Doesn't seem to handle dead guards/empty slots. I noticed you manually iterated over the guards, which the GuardDataReader class is designed to do to simplify things. So check it out if you haven't already

Looking forward to any future scripts! (Been using the RNG seed script lately and I must say, it's really useful )

[Whiteted]

• Oops that's an obvious oversight, I was only trying it on one level. I'll have a look at how to generalise it, but the values in your metadata of the player data don't account for bond wobbling about
• Wait I can play not in 320x240 resolution? I'll definitely look at that map code.
• Yeah the guards are really just an example, I think I even had a marker up in the air somewhere. That said I actually copied code from the guardDataReader rather than using it, which might be some sort of cardinal sin

There's also a fun glitch if you look right down it shows all the points in a spiral.. I'm not quite sure what's going on there lol.

Thanks again for the great feedback, I'll try and make those changes now. I'm glad you like the RNG seed script

Also do you know how I can find out if I'm on a lag frame?

[Whiteted]

Metadata watch generator thing

I've got to keep myself off the streets so I've been writing more lua scripts. This is just a little one for taking a table of metadata and a base address and producing a .wch watch file for bizhawk, with notes naming the individual addresses.

It's main feature is gap filling: If the checkbox is checked it will give you a contiguous block of addresses, labelling those which do not have names by "[UNKNOWN] 0x_" where _ is the offset from the base address. I've been using this to extend Henrik's metadata tables slightly.

https://github.com/whiteted-strats/Minor_lua/tree/master/metadata_watch

[Whiteted]

HUD Matt (v3)
https://github.com/whiteted-strats/GE_HUD



Main update is it's now nicely encapsulated in the _core, _lib and _clip files. These need access to some of Henrik's Data/GE/* files.
The example (HUD_all_guards) shows the collision diamonds & circles for guards who are 'visible' (in a loaded room?), and shows that it's now very easy to use.

It's going in the right direction since it now actually reads a matrix which moves from the global world to the 'coordinate world', with the camera looking down the z axis. The big problem is it's often not reading the right matrix: In lag it performs awfully. If anyone wants to fix this so I don't have to there is the matrix_finder.lua script. It also only supports aztec & s1 atm coz I couldn't be bothered to find more matrices.

Fun fact: The camera actually does have some roll during normal gameplay.

[Whiteted]

As featured in my SA throw strat idea post:
https://github.com/whiteted-strats/Caverns_October/blob/master/HUD_timed_mines.lua



A really good use of the Matrix based HUD, giving a perfect HUD when you are standing still. May have lag issues, atm kinda sorted out by having the D key show/hide the display. Could be useful on other levels, particularly some R-leaning ones?

[Wodahs-Reklaw]

Wow sick hud. I will have to give it a try.

[Whiteted]

Thanks instructions for using it are in the readme / comments in the top of the file. Let me know if you have any issues.

I also just thought that a handy script to have would be one which performs an r-lean for you (in the specified direction), possibly automatically making repeated attempts. I'll edit this post when I start making it, but I don't know much about r-leans so if someone else wants to write it that would be great :P

[Whiteted]

Last shot HUD (and HUD_Matt v3.1)

https://github.com/whiteted-strats/GE_HUD/blob/master/HUD_last_shot.lua
Mostly just demonstrating my find: the origin & direction of the last 'live' bullet fired by a guard (addition to Henrik's guardData.lua). Handy for seeing which way guards behind you shot (such as for my 3 AR boosts on caverns idea: turns out my TAS did get the 3rd AR boost nearby, so may just have been unlucky).

I'll hopefully use this soon to look at odds of boosts and how to maximise them. Also to show it's a myth (probably) that looking down reduces your chances of getting hit and similarly that it's sort of a myth (maybe) that strafing does too (I'm pretty sure Bond's collision box is square, but with the corners always facing north/east/south/west regardless of which way you face).

The only thing so far is that guards with two guns only seem to fire out of the gun in their right hand (this seems to be the case with Jaws too):

Also the general HUD_Matt has been updated quite a bit, and now has matrices for all levels except Control (minor mystery where that one is)