Part of mobile device support:
https://github.com/kanaka/noVNC/issues/48
The Display object is redefined as a larger display region with
an equal or smaller visible viewport. The size of the full display
region is set/changed using resize(). The viewport is set/changed
using viewportChange().
All exposed routines that draw on the display now take coordinates
that are absolute (relative to the full display region). For example,
the result of fillRect(100, 100, 10, 10, [255,0,0]) will appear in the
canvas at (0,0) if the viewport is set to (100,100).
Details:
- Move the generic part of the viewport code from tests/viewport.html
into include/display.
- Add two new routines to the Display interface:
- viewportChange(deltaX, deltaY, width, height)
- This adjusts the position of the visible viewport and/or the
size of the viewport.
- deltaX and deltaY specify how the position of the viewport
should be shifted. The position of the viewport is clamped
to the full region size (i.e. cannot outside the display
region).
- The clean and dirty regions of the display are updated based
on calls to this routine. For example, if the viewport width
is increased, then there is now a dirty box on the right
side of the viewport. Another example, if the viewport is
shifted down and to the left over the display region, there
are now two dirty boxes: one on the left side and one
on the bottom of the viewport.
- getCleanDirtyReset()
- This returns an object with the clean box and a list of
dirty boxes (that need to be redrawn).
{'cleanBox':
{'x': x, 'y': y, 'w': w, 'h': h},
'dirtyBoxes':
[{'x': x, 'y': y, 'w': w, 'h': h}, ...]
}
- The coordinates in the clean and dirty boxes are absolute
coordinates (relative to the full display region) but they
are clipped to the visible viewport.
- Calling this function also resets the clean rectangle to be
the whole viewport (i.e. nothing visible needs to be redrawn
dirty) so the caller of this routine is responsible for
redrawing any
Tested on iOS (iPhone and iPad).
The viewport is correctly clipped to the screen/browser size and
resizing works correctly.
This uses the CSS3 Flexible Box Layout model.
Tested with an iPad 2.
This example shows a 400x200 viewport of an 800x400 display.
It tries to be intelligent about how much it redraws. It copies what
it can, and then when the user releases the mouse, it redraws the
"dirty" areas that were newly revealed.
First crack at supporting touch screen for devices like Android and
iOS tablets. Part of https://github.com/kanaka/noVNC/issues/48.
This change detects touch screen support and uses the touchstart,
touchmove, touchend events in place of the normal mouse events.
In order to support middle and right mouse clicks, if the device is
a touch device, then three toggle buttons are added to the UI
representing the left, middle and right mouse buttons. These select
which mouse button will be sent when the screen is touched. All the
buttons can be toggled off, in which case then the touch events only
move the mouse cursor rather than sending a mouse down and mouse up
for touchstart and touchend events respectively. This allows fairly
full control with the mouse on touch screens.
Instead of R,G,B (red-shift of 0, green-shift of 8, and blue-shift
of 16), use the default ordering of B,G,R (red-shift of 16, green-shift of 8, and blue-shift
of 0) that tightvncserver uses (and that VMWare's VNC server seems to
require). Also, warn in the console if the server does not default to
the new format.
Fix the tests/canvas.html test. This is a general fix with regards to
the rename/refactor of canvas.js into display.js and not specific to
the color re-ordering.
This is part of addressing issue #21 - non-US keyboard layouts.
There are several challenges when dealing with keyboard events:
- The meaning and use of keyCode, charCode and which depends on
both the browser and the event type (keyDown/Up vs keyPress).
- We cannot automatically determine the keyboard layout
- The keyDown and keyUp events have a keyCode value that has not
been translated by modifier keys.
- The keyPress event has a translated (for layout and modifiers)
character code but the attribute containing it differs. keyCode
contains the translated value in WebKit (Chrome/Safari), Opera
11 and IE9. charCode contains the value in WebKit and Firefox.
The which attribute contains the value on WebKit, Firefox and
Opera 11.
- The keyDown/Up keyCode value indicates (sort of) the physical
key was pressed but only for standard US layout. On a US
keyboard, the '-' and '_' characters are on the same key and
generate a keyCode value of 189. But on an AZERTY keyboard even
though they are different physical keys they both still
generate a keyCode of 189!
- To prevent a key event from propagating to the browser and
causing unwanted default actions (such as closing a tab,
opening a menu, shifting focus, etc) we must suppress this
event in both keyDown and keyPress because not all key strokes
generate on a keyPress event. Also, in WebKit and IE9
suppressing the keyDown prevents a keyPress but other browsers
still generated a keyPress even if keyDown is suppressed.
For safe key events, we wait until the keyPress event before
reporting a key down event. For unsafe key events, we report a key
down event when the keyDown event fires and we suppress any further
actions (including keyPress).
In order to report a key up event that matches what we reported
for the key down event, we keep a list of keys that are currently
down. When the keyDown event happens, we add the key event to the
list. If it is a safe key event, then we update the which attribute
in the most recent item on the list when we received a keyPress
event (keyPress should immediately follow keyDown). When we
received a keyUp event we search for the event on the list with
a matching keyCode and we report the character code using the value
in the 'which' attribute that was stored with that key.
For character codes above 255 we use a character code to keysym lookup
table. This is generated using the util/u2x11 script contributed by
Colin Dean (xvpsource.org).
API change: for intergrators that explicitly include the Javascript
files (that do not use include/vnc.js)js, include/input.js is a new
file that must also be included.
The mouse and keyboard handling could be useful on its own so split it
out into a Keyboard and Mouse class in include/input.js.
This refactoring is preparation to deal with issue #21 - non-US
keyboard layouts.
Fix mouse button mapping in IE9. All browsers have converged on
a standard left=0, middle=1, right=2 ... all except IE that is.
Add html5 doctype to tests.
In vnc_perf test, use do_test instead of start for function name since
start is a keyword in IE.
In error about Flash give a link to Adobe's download page.
https://github.com/kanaka/websockify is now the canonical location of
websockify (formerly wsproxy). A copy of the python version is kept
here for backwards compatibility and ease-of-use. The other versions
and related test scripts are in websockify.
- add dup2 functionality. This requires adding a ref cnt to the
_WS_connections structure so that we only free the structure once
all dup'd referenced are closed. Also, refactor malloc and free of
connection structure into _WS_alloc and _WS_free.
- allow select to accept a NULL timeout value which means sleep
forever instead of segfaulting.
- fix some compile warnings related to ppoll definition.
- move some WebSockets related html test pages into utils and symlink
them from tests.
Moved websocket.py code into a class WebSocketServer. WebSockets
server implementations will sub-class and define a handler() method
which is passed the client socket after. Global variable settings have been
changed to be parameters for WebSocketServer when created.
Subclass implementations still have to handle queueing and sending but
the parent class handles everything else (daemonizing, websocket
handshake, encode/decode, etc). It would be better if the parent class
could handle queueing and sending. This adds some buffering and
polling complexity to the parent class but it would be better to do so
at some point. However, the result is still much cleaner as can be
seen in wsecho.py.
Refactored wsproxy.py and wstest.py (formerly ws.py) to use the new
class. Added wsecho.py as a simple echo server.
- rename tests/ws.py to utils/wstest.py and add a symlink from
tests/wstest.py
- rename tests/ws.html to tests/wstest.html to match utils/wstest.py.
- add utils/wsecho.py
- add tests/wsecho.html which communicates with wsecho.py and simply
sends periodic messages and shows what is received.
Rename the $() selector to $D() so that it doesn't collide with
the jQuery name.
The API change is that the 'target' option for Canvas and RFB objects
must now be a DOM Canvas element. A string is no longer accepted
because this requires that a DOM lookup is done and the Canvas and RFB
should have no UI code in them. Modularity.
- util.js that contains essential functions
- webutils.js that contains the GUI utility function.js
this helps to include noVNC in other project, especially Cappuccino Application
i
- include/rfb.js: Keep track of the number of rects of each encoding
type and print them out when we close a connection (if 'info'
logging level).
- tests/vnc_perf.html: first pass at a noVNC based performance
benchmark.
- utils/wsproxy.py: Fix the output of the record filename.
- include/canvas.js: When 'debug' logging, show browser detection
values.
- test/canvas.html: Only restore the canvas to it's starting state if
the logging level is not 'debug'.
- wsproxy.py: Append the session number to the record filename so that
multiple sessions don't stomp on each other.
Generally, most servers send hextile updates as single updates
containing many rects. Some servers send hextile updates as many small
framebuffer updates with a few rects each (such as QEMU). This latter
cases revealed that shifting off the beginning of the receive queue
(which happens after each hextile FBU) performs poorly.
This change switches to using an indexed receive queue (instead of
actually shifting off the array). When the receive queue has grown to
a certain size, then it is compacted all at once.
The code is not as clean, but this change results in more than 2X
speedup under Chrome for the pessimal case and 10-20% in firefox.
Turns out when Windows is running in QEMU and a window scroll happens,
there are lots of little hextile rects sent. This is slow in noVNC.
- Some recording/playback improvement.
- Add test harness to drive playback of recordings.
- By pulling off the rect header in one chunk we get a 3X speedup in
Chrome and a 20% speedup in firefox (specifically for the scroll
test).
- Also, get rid of some noise from creating timers for handle_message.
Check to make sure there isn't already a pending timer first.
New API:
To use the RFB object, you now must instantiate it (this allows more
than one instance of it on the same page).
rfb = new RFB(settings);
The 'settings' variable is a namespace that contains initial default
settings. These can also be set and read using 'rfb.set_FOO()' and
'rfb.get_FOO()' where FOO is the setting name. The current settings
are (and defaults) are:
- target: the DOM Canvas element to use ('VNC_canvas').
- encrypt: whether to encrypt the connection (false)
- true_color: true_color or palette (true)
- b64encode: base64 encode the WebSockets data (true)
- local_cursor: use local cursor rendering (true if supported)
- connectTimeout: milliseconds to wait for connect (2000)
- updateState: callback when RFB state changes (none)
- clipboardReceive: callback when clipboard data received (none)
The parameters to the updateState callback have also changed. The
function spec is now updateState(rfb, state, oldstate, msg):
- rfb: the RFB object that this state change is for.
- state: the new state
- oldstate: the previous state
- msg: a message associate with the state (not always set).
The clipboardReceive spec is clipboardReceive(rfb, text):
- rfb: the RFB object that this text is from.
- text: the clipboard text received.
Changes:
- The RFB and Canvas namespaces are now more proper objects. Private
implementation is no longer exposed and the public API has been made
explicit. Also, instantiation allows more than one VNC connection
on the same page (to complete this, DefaultControls will also need
this same refactoring).
- Added 'none' logging level.
- Removed automatic stylesheet selection workaround in util.js and
move it to defaultcontrols so that it doesn't interfere with
intergration.
- Also, some major JSLinting.
- Fix input, canvas, and cursor tests to work with new model.
The following API changes may affect integrators:
- Settings have been moved out of the RFB.connect() call. Each
setting now has it's own setter function: setEncrypt, setBase64,
setTrueColor, setCursor.
- Encrypt and cursor settings now default to on.
- CSS changes:
- VNC_status_bar for input buttons switched to a element class.
- VNC_buttons split into VNC_buttons_right and
VNC_buttons_left
- New id styles for VNC_settings_menu and VNC_setting
Note: the encrypt, true_color and cursor, logging setting can all be
set on load using query string variables (in addition to host, port
and password).
Client cursor (cursor pseudo-encoding) support has been polished and
activated.
The RFB settings are now presented as radio button list items in
a drop-down "Settings" menu when using the default controls.
Also, in the settings menu is the ability to select between alternate
style-sheets.
Cookie and stylesheet selection support added to util.js.
Interestingly it turns out that using the native base64 routines does
not improve performance. Likely because the actual time is in
marshalling/unmarshalling between strings and arrays (and associated
garbage collection overhead) which has to be done either way.
Looks like disabling web-socket-js debug messages by default that we
get a minor speedup.
Python proxy should support both 75 and 76 (00) modes. Also, update ws
test to more reliably hit the WebSockets ordering/drop issue.
Also add a wsencoding test client/server program to test send a set of
values between client and server and vice-versa to test encodings.
Not turned on by default.
Add support for encode/decode of UTF-8 in the proxy. This leverages
the browser for decoding the WebSocket stream directly instead of
doing base64 decode in the browser itself.
Unfortunately, in Chrome this has negligible impact (round-trip time
is increased slightly likely due to extra python processing).
In firefox, due to the use of the flash WebSocket emulator the
performance is even worse. This is because it's really annoying to get
the flash WebSocket emulator to properly decode a UTF-8 bytestream.
The problem is that the readUTFBytes and readMultiByte methods of an
ActionScript ByteArray don't treat 0x00 correctly. They return
a string that ends at the first 0x00, but the index into the ByteArray
has been advanced by however much you requested.
This is very silly for two reasons: ActionScript (and Javascript)
strings can contain 0x00 (they are not null terminated) and second,
UTF-8 can legitimately contain 0x00 values. Since UTF-8 is not
constant width there isn't a great way to determine if those methods
in fact did encounter a 0x00 or they just read the number of bytes
requested.
Doing manual decoding using readUTFByte one character at a time slows
things down quite a bit. And to top it all off, those methods don't
support the alternate UTF-8 encoding for 0x00 ("\xc0\x80"). They also
just treat that encoding as the end of string too.
So to get around this, for now I'm encoding zero as 256 ("\xc4\x80")
and then doing mod 256 in Javascript. Still doesn't result in much
benefit in firefox.
But, it's an interesting approach that could use some more exploration
so I'm leaving in the code in both places.
- By dereferencing the 'data' field of the imageData object before the
loop, the hextile performance on Chrome is down to 140ms or so for
a full 800x600 update. Still have to fall back to Canvas operations
for firefox.
- Fix RQ empty after reorder bug.
Joel Martin
Antoine Mercadal
Joel Martin