1.2.1.13 HTTP Net Module
The HTTP Net Web Server module allows a development board to act as a Web server. This facilitates an easy method to view status information and control applications using any standard Web browser. It uses the Networking Presentation Layer to integrate with an external encryption services provider (usually wolfSSL) allowing for secure connections.
Note: The HTTP_NET server supports encrypted communication by using the Networking Presentation layer to integrate an external encryption services provider (usually wolfSSL) for TLS support.
Abstraction Model
![HTTP NET Abstraction Model](GUID-20ECEDDE-25E9-4932-A1D7-7CB916C7A43E-low.png)
HTTP Net Features
The HTTP Net module is a HTTP server module that works using the Networking Presentation Layer of the MPLAB Harmony framework. This allows it to support both encrypted and plain text connections.
Some of the key features of the HTTP Net module include:
The updated module implements HTTP 1.1 persistent connections by default, using the “Transfer-Encoding: chunked” HTTP header. It can be configured to operate with closing the connections like the previous HTTP server.
The application has to dynamically register its HTTP processing functions with the HTTP Net server using the TCPIP_HTTP_NET_UserHandlerRegister call. The HTTP Net server no longer relies on well known function names.
The HTTP Net server supports either secure or plain text connections, but not both simultaneously. The specification of a secure or non-secure connection can be easily done by:
Selecting the server listening port: 80 or 443
Using the module configuration flags
The HTTP Net module reports run-time events that could be caught by the user application using the “eventReport” registered function that carries a TCPIP_HTTP_NET_EVENT_TYPE enumerated type
Dynamic Variables Processing
Parsing of the dynamic variables is done at run-time. The web pages could be changed at run-time, without the need for rebuilding the HTTP Net server. This allows the TCP/IP Stack to run as a library and only the application code to be changed accordingly. The application is required to register a “dynamicPrint” with the HTTP Net server. This function will be called at run-time to process the dynamic variables.
The supported syntax is var_name(param1, param2, …). When parsing a string like this within a web page, the HTTP Net server will invoke the “dynamicPrint” function using a TCPIP_HTTP_DYN_VAR_DCPT data structure that specifies the dynamic variable name, its number and type of parameters as well as each parameter value.
For example, the variable myVariable(2,6) will generate the “dynamicPrint” call with the following parameters:
varDcpt.dynName = "myVariable";
varDcpt.nArgs = 2;
varDcpt.dynArgs->argType = TCPIP_HTTP_DYN_ARG_TYPE_INT32;
varDcpt.dynArgs->argInt32 = 2;
(varDcpt.dynArgs + 1)->argType = TCPIP_HTTP_DYN_ARG_TYPE_INT32;
(varDcpt.dynArgs + 1)->argInt32 = 6;
String and int32_t variable types are currently supported.
The application needs to return a TCPIP_HTTP_DYN_PRINT_RES result specifying if it’s done with processing or it needs to be called again.
Applications no longer have direct access to the underlying transport socket for a HTTP connection. All of the data write operations need to go through the required API function: TCPIP_HTTP_NET_DynamicWrite. This function actually just attaches the data to be written to the connection, using data descriptors. The actual output occurs only when all data is gathered and the data size is known.
Buffers used in a TCPIP_HTTP_NET_DynamicWrite need to be persistent until the moment when the output is generated. RAM buffers are supported by using a “dynamicAck” (TCPIP_HTTP_NET_DynAcknowledge) type function. Once the output operation is completed the HTTP Net module will call back into the application indicating that the corresponding buffer is no longer used and can be freed/reused.
In situations where the dynamic variable print function needs to perform additional write operations, or simply needs to be called again, it must a special result code: TCPIP_HTTP_DYN_PRINT_RES_PROCESS_AGAIN/TCPIP_HTTP_DYN_PRINT_RES_AGAIN. Typically this is used when outputting large amounts of data that cannot fit into one single buffer write operation or when the data is not available all at once.
Note: The parsing of the files for dynamic content is done at run time. Therefore when generating the image, files that contain dynamic content should not be compressed!
Include Files Processing
The HTTP Net module dynamically parses the web pages and the file inclusion is done by the module itself, completely transparent for the application.
The included files can contain dynamic variables that are parsed at run time.
File Processing
File processing consists of the following:
An included file can include other files that have dynamic variables
Application can include a file as part of its dynamic variable processing using the TCPIP_HTTP_NET_ConnectionFileInclude()API. That file can include other files.
The only limit to the depth of inclusion is the memory resources and the TCPIP_HTTP_NET_MODULE_CONFIG:: maxRecurseLevel setting. A TCPIP_HTTP_NET_EVENT_DEPTH_ERROR event will be generated if the maximum depth is exceeded at run-time.
SSI Processing
At run-time, the HTTP Net server processes a subset of the Server Side Includes (SSI) commands. SSI is a portable way to add small amounts of dynamic content on web pages in a standard way that is supported by most HTTP servers. Currently the #include, #set and #echo commands are supported. Other commands will be eventually added. This allows a better compatibility with other existing HTTP servers and is targeted at the removal of proprietary extensions.
Whenever executing an SSI command within a web page, the HTTP Net server will notify the application by using the registered SSI callback: ssiNotify. This callback contains a TCPIP_HTTP_SSI_NOTIFY_DCPT pointer to a structure describing the SSI context needed for processing:
File name the SSI command belongs to
The SSI command line
Number of SSI attributes and their values, etc.
The application can do any modifications it chooses to and returns a value instructing the HTTP Net server if the processing of the command is necessary or not.
SSI include Command
The SSI Include command allows the dynamic inclusion of a file. The supported syntax is:
<!--#include virtual="file_name" --> or <!--#include file="file_name" -->.
Currently the arguments for both “virtual” and “file” commands are passed unaltered to the SYS_FS, so they behave identically. However it is recommended that the original SSI significance for these keywords should be maintained:
Use “virtual” for specifying a URL relative to the document being server
Use “file” for a file path, relative to the current directory (it cannot be an absolute path) The inc:file_name keyword is maintained for backward compatibility. However the SSI include command should be preferred.
SSI set Command
The SSI set command allows to dynamically set a SSI variable value. The supported syntax is:
<!--#set var="v_name" value="v_value" -->.
String or integer variables are supported. Variable reference is also supported:
<!--#set var="n_name" value="$otherVar" -->.
This command will create or update the variable n_name to have the value of the variable otherVar, if it exists.
A new SSI variable will be created if a variable having the name v_name does not exist. If the variable v_name already exists, it will have its value updated as the result of this command.
An existing variable can be deleted using the empty value set command:
<!--#set var="v_name" value="" -->.
SSI echo Command The SSI echo command allows the dynamic print of a SSI variable. The supported syntax is:
<!--#echo var="v_name" -->.
If the application ssiNotify exists, the HTTP Net server will call it and the application may choose to change the current value dynamically. If ssiNotify returns false, the HTTP Net server will display the current value of the variable v_name as part of the current page.
The SSI API can be used to evaluate or change the current value of the SSI variables:
TCPIP_HTTP_NET_SSIVariableGet
TCPIP_HTTP_NET_SSIVariableSet
TCPIP_HTTP_NET_SSIVariableDelete
The maximum number of SSI variables is under the control of the application by using the configuration parameter: TCPIP_HTTP_NET_SSI_VARIABLES_NUMBER.
mpfs2.jar Utility
Dynamic Variable Processing
The HTTP Net server does not use the DynRecord.bin and FileRcrd.bin files for processing the dynamic variables. The parsing and processing of dynamic variables is done at run-time. Therefore, the mpfs2.jar utility is no longer necessary, other than to generate a mpfs_img2.c image file for the NVM MPFS image.
Application Custom Support Files
The application custom support files generated with MHC have new names: custom_http_net_app.c and http_net_print.c. However, the http_net_print.c file is not generated by the mpfs2 utility and is maintained only for easy comparison between HTTP and HTTP Net application processing. The custom_http_net_app.c file is entirely generated using a MHC template file and it is not dynamically updated in any way by the mpfs2 utility.
Generated File Name
Currently the name of the generated file for MPFS image is maintained unchanged: mpfs_img2.c.
MPFS Image Generation for Internal NVM Storage
The mpfs2 utility can still be used to generate the MPFS image for internal NVM storage. It can also be useful because it parses the web pages and comes out with the http_print.c file that contains the list of TCPIP_HTTP_Print functions. This can be helpful in gathering info about the dynamic variables that are contained within the web pages.
Note: For demonstration applications that use SSI, the file inclusion is now done in a standard way using HTML (i.e., .htm) files. Therefore, when generating the image, *.htm must be added to Advanced Settings > Do Not Compress.
Library Interface
Name | Description |
---|---|
Macros | |
TCPIP_HTTP_NET_CACHE_LEN | Max lifetime (sec) of static responses as string |
TCPIP_HTTP_NET_CONFIG_FLAGS | Define the HTTP module configuration flags Use 0 for default See HTTP_MODULE_FLAGS definition for possible values |
TCPIP_HTTP_NET_COOKIE_BUFFER_SIZE | Size of the buffer used for sending the cookies to the client. Should be able to accommodate the longest cookie response. Otherwise the cookies will be truncated. |
TCPIP_HTTP_NET_DEFAULT_FILE | Indicate what HTTP file to serve when no specific one is requested |
TCPIP_HTTP_NET_DYNVAR_ARG_MAX_NUMBER | maximum number of arguments for a dynamic variable |
TCPIP_HTTP_NET_DYNVAR_DESCRIPTORS_NUMBER | how many buffers descriptors for dynamic variable processing they are independent of the HTTP connection number all the HTTP connections use from the dynamic descriptors pool |
TCPIP_HTTP_NET_DYNVAR_MAX_LEN | maximum size for a complete dynamic variable: name + args must be <= TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_SIZE! If it is much larger than needed then inefficiency occurs when reading data from the file and then discarding it because a much larger than needed data buffer was read |
TCPIP_HTTP_NET_FILE_UPLOAD_ENABLE | Configure MPFS over HTTP updating Comment this line to disable updating via HTTP |
TCPIP_HTTP_NET_FILE_UPLOAD_NAME | This is macro TCPIP_HTTP_NET_FILE_UPLOAD_NAME. |
TCPIP_HTTP_NET_FIND_PEEK_BUFF_SIZE | Size of the peek buffer to perform searches into. If the underlying transport layer supports offset peak operation with a offset, the value could be smaller (80 characters, for example); otherwise, a one time peek is required and the buffer should be larger - recommended to be close to the size of the socket RX buffer. |
TCPIP_HTTP_NET_MAX_CONNECTIONS | Maximum numbers of simultaneous supported HTTP connections. |
TCPIP_HTTP_NET_MAX_DATA_LEN | Define the maximum data length for reading cookie and GET/POST arguments (bytes) |
TCPIP_HTTP_NET_MAX_HEADER_LEN | The length of longest header string that can be parsed |
TCPIP_HTTP_NET_MAX_RECURSE_LEVEL | The maximum depth of recursive calls for serving a web page: no dynvars files: 1; file including a file: 2 ; if the include file includes another file: +1 ; if a dyn variable: +1; Default value is 3; |
TCPIP_HTTP_NET_RESPONSE_BUFFER_SIZE | Size of the buffer used for sending the response messages to the client. Should be able to accommodate the longest server response: Default setting should be 300 bytes |
TCPIP_HTTP_NET_SKT_RX_BUFF_SIZE | Define the size of the RX buffer for the HTTP socket Use 0 for default TCP socket value The default recommended value for high throughput is > 2MSS (3 KB). The performance of a socket is highly dependent on the size of its buffers so it's a good idea to use as large as possible buffers for the sockets that need high throughput. |
TCPIP_HTTP_NET_SKT_TX_BUFF_SIZE | Define the size of the TX buffer for the HTTP socket Use 0 for default TCP socket value The default recommended value for high throughput is > 2MSS (3 KB). The performance of a socket is highly dependent on the size of its buffers so it's a good idea to use as large as possible buffers for the sockets that need high throughput. |
TCPIP_HTTP_NET_TASK_RATE | The HTTP task rate, ms The default value is 33 milliseconds. The lower the rate (higher the frequency) the higher the module priority and higher module performance can be obtained The value cannot be lower than the TCPIP_STACK_TICK_RATE. |
TCPIP_HTTP_NET_TIMEOUT | Max time (sec) to await for more HTTP client data in an active connection state before timing out and disconnecting the socket |
TCPIP_HTTP_NET_USE_AUTHENTICATION | Enable basic authentication support |
TCPIP_HTTP_NET_USE_COOKIES | Enable cookie support |
TCPIP_HTTP_NET_USE_POST | Define which HTTP modules to use If not using a specific module, comment it to save resources Enable POST support |
TCPIP_HTTP_NET_CHUNK_RETRIES | retry limit for allocating a chunk from the pool If more retries are not successful the operation will be aborted |
TCPIP_HTTP_NET_CHUNKS_NUMBER | number of chunks that are created It depends on the TCPIP_HTTP_NET_MAX_RECURSE_LEVEL and on the number of connections Maximum number should be TCPIP_HTTP_NET_MAX_CONNECTIONS * TCPIP_HTTP_NET_MAX_RECURSE_LEVEL i.e. TCPIP_HTTP_NET_MODULE_CONFIG::nConnections * TCPIP_HTTP_NET_MODULE_CONFIG::nChunks All the chunks are in a pool and are used by all connections |
TCPIP_HTTP_NET_DYNVAR_PROCESS | This symbol enables the processing of dynamic variables Make it evaluate to false (0) if dynamic variables are not needed All the following symbols referring to dynamic variables are relevant only when TCPIP_HTTP_NET_DYNVAR_PROCESS != 0 |
TCPIP_HTTP_NET_DYNVAR_PROCESS_RETRIES | retry limit for a dynamic variable processing ths puts a limit on the number of times a dynamic variable "dynamicPrint" function can return TCPIP_HTTP_DYN_PRINT_RES_AGAIN/TCPIP_HTTP_DYN_PRINT_RES_PROCESS_AGAIN and avoids having the HTTP code locked up forever. If more retries are attempted the processing will be considered done and dynamicPrint function will not be called again |
TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_RETRIES | Retry limit for allocating a file buffer from the pool. If more retries are not successful the operation will be aborted. |
TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_SIZE | Size of the buffer used for processing HTML, dynamic variable and binary files. For dynamic variable files it should be able to accommodate the longest HTML line size, including CRLF! |
TCPIP_HTTP_NET_FILE_PROCESS_BUFFERS_NUMBER | Number of file buffers to be created; These buffers are used to store data while file processing is done They are organized in a pool Each file being processed needs a file buffer and tries to get it from the pool If a buffer is not available, the HTTP conenction will wait for one to become available. Once the file is done the file buffer is released and could be used by a different file The number depends on the number of files that are processed in parallel To avoid deadlock the number should be >= than the number of... more |
TCPIP_HTTP_NET_FILENAME_MAX_LEN | maximum size of a HTTP file name with the path removed from the file name one extra char added for the string terminator |
TCPIP_HTTP_NET_SSI_ATTRIBUTES_MAX_NUMBER | maximum number of attributes for a SSI command most SSI commands take just one attribute/value pair per line but multiple attribute/value pairs on the same line are allowed where it makes sense |
TCPIP_HTTP_NET_SSI_CMD_MAX_LEN | maximum size for a SSI command line: command + attribute/value pairs must be <= TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_SIZE! If it is much larger than needed then inefficiency occurs when reading data from the file and then discarding it because a much larger than needed data buffer was read |
TCPIP_HTTP_NET_SSI_ECHO_NOT_FOUND_MESSAGE | message to echo when echoing a not found variable |
TCPIP_HTTP_NET_SSI_PROCESS | This symbol enables the processing of SSI commands Make it evaluate to false (0) if SSI commands are not needed All the following symbols referring to SSI commands are relevant only when TCPIP_HTTP_NET_SSI_PROCESS != 0 |
TCPIP_HTTP_NET_SSI_STATIC_ATTTRIB_NUMBER | number of static attributes associated to a SSI command if the command has more attributes than this number the excess will be allocated dynamically |
TCPIP_HTTP_NET_SSI_VARIABLE_NAME_MAX_LENGTH | maximum length of a SSI variable name any excess characters will be truncated Note that this can result in multiple variables being represented as one SSI variable |
TCPIP_HTTP_NET_SSI_VARIABLE_STRING_MAX_LENGTH | maximum size of a SSI string variable value any excess characters will be truncated Note that the variable value requires SSI storage that's allocated dynamically Also, this value determines the size of an automatic (stack) buffer when the variable is echoed. If this value is large, make sure you have enough stack space. |
TCPIP_HTTP_NET_SSI_VARIABLES_NUMBER | maximum number of SSI variables that can be created at run time These variables are stored in an internal hash. For max. efficiency this number should be a prime. |
TCPIP_HTTP_NET_CONNECTION_TIMEOUT | Persistent connection idle timeout, in seconds If a persistent connection is idle for more that this timeout the server will close it. Usually the client closes the connection. Using this timeout value the server can close the its side of the connection to avoid all connections to remain indefinitely open if the client misbehaves. Use 0 to never timeout. The timeout value has to be <= 32767 seconds. |
HTTP_NET_CONFIG_H | This is macro HTTP_NET_CONFIG_H. |
TCPIP_HTTP_NET_FREE_FUNC | HTTP corresponding deallocation function, free style This is the function the HTTP will call for freeing the allocated memory Use standard C library 'free' or 0 as a default If it's 0, HTTP will use the allocation functions passed in at the stack initialization |
TCPIP_HTTP_NET_MALLOC_FUNC | HTTP allocation function, malloc style This is the function the HTTP will call to allocate memory needed for: dynamic variables, SSI or file uploads. Use standard C library 'malloc' or 0 as a default If it's 0, HTTP will use the allocation functions passed in at the stack initialization |
Functions | |
TCPIP_HTTP_NET_ConnectionByteCountDec | Decrements the connection byte count. |
TCPIP_HTTP_NET_ConnectionByteCountSet | Sets how many bytes are available to be read. |
TCPIP_HTTP_NET_ConnectionCallbackPosSet | Sets the callback position indicator. |
TCPIP_HTTP_NET_ConnectionDiscard | Discards any pending data in the connection RX buffer. |
TCPIP_HTTP_NET_ConnectionFlush | Immediately transmits all connection pending TX data. |
TCPIP_HTTP_NET_ConnectionHasArgsSet | Sets whether there are get or cookie arguments. |
TCPIP_HTTP_NET_ConnectionIsAuthorizedSet | Sets the authorized state for the current connection. |
TCPIP_HTTP_NET_ConnectionNetHandle | Returns the network handle of the current connection. |
TCPIP_HTTP_NET_ConnectionPeek | Reads a specified number of data bytes from the connection RX buffer without removing them from the buffer. |
TCPIP_HTTP_NET_ConnectionPostSmSet | Set the POST state machine state. |
TCPIP_HTTP_NET_ConnectionRead | Reads an array of data bytes from a connection's RX buffer. |
TCPIP_HTTP_NET_ConnectionReadBufferSize | Returns the size of the connection RX buffer. |
TCPIP_HTTP_NET_ConnectionReadIsReady | Determines how many bytes can be read from the connection RX buffer. |
TCPIP_HTTP_NET_ConnectionStatusSet | Sets HTTP status. |
TCPIP_HTTP_NET_ConnectionStringFind | Helper to find a string of characters in an incoming connection buffer. |
TCPIP_HTTP_NET_ConnectionUserDataSet | Sets the user data parameter for the current connection. |
TCPIP_HTTP_NET_DynamicWrite | Writes a data buffer to the current connection |
TCPIP_HTTP_NET_DynamicWriteString | Helper for writing a string within a dynamic variable context. |
TCPIP_HTTP_NET_Task | Standard TCP/IP stack module task function. |
TCPIP_HTTP_NET_URLDecode | Parses a string from URL encoding to plain text. |
TCPIP_HTTP_NET_UserHandlerDeregister | Deregisters a previously registered HTTP user handler. |
TCPIP_HTTP_NET_UserHandlerRegister | Registers a user callback structure with the HTTP server |
TCPIP_HTTP_NET_ActiveConnectionCountGet | Gets the number of active (and inactive) connections. |
TCPIP_HTTP_NET_ArgGet | Locates a form field value in a given data array. |
TCPIP_HTTP_NET_ConnectionByteCountGet | Returns how many bytes are available to be read. |
TCPIP_HTTP_NET_ConnectionCallbackPosGet | Returns the callback position indicator. |
TCPIP_HTTP_NET_ConnectionDataBufferGet | Returns pointer to connection general purpose data buffer. |
TCPIP_HTTP_NET_ConnectionFileGet | Get handle to current connection's file. |
TCPIP_HTTP_NET_ConnectionHasArgsGet | Checks whether there are get or cookie arguments. |
TCPIP_HTTP_NET_ConnectionIsAuthorizedGet | Gets the authorized state for the current connection. |
TCPIP_HTTP_NET_ConnectionPostSmGet | Get the POST state machine state. |
TCPIP_HTTP_NET_ConnectionSocketGet | Get the socket for the current connection. |
TCPIP_HTTP_NET_ConnectionStatusGet | Gets HTTP status. |
TCPIP_HTTP_NET_ConnectionUserDataGet | Gets the user data parameter for the current connection. |
TCPIP_HTTP_NET_ConnectionPostNameRead | Reads a name from a URL encoded string in the network transport buffer. |
TCPIP_HTTP_NET_ConnectionPostValueRead | Reads a value from a URL encoded string in the network transport buffer. |
TCPIP_HTTP_NET_ConnectionDataBufferSizeGet | Returns the size of the connection general purpose data buffer. |
TCPIP_HTTP_NET_ConnectionDynamicDescriptors | Returns the number of dynamic variable descriptors |
TCPIP_HTTP_NET_SSIVariableDelete | Function to delete an SSI variable. |
TCPIP_HTTP_NET_SSIVariableGet | Function to get access to an existing SSI variable. |
TCPIP_HTTP_NET_SSIVariableGetByIndex | Function to get access to an existing SSI variable. |
TCPIP_HTTP_NET_SSIVariableSet | Function to set an SSI variable. |
TCPIP_HTTP_NET_SSIVariablesNumberGet | Function to get the number of the current SSI variables. |
TCPIP_HTTP_NET_ConnectionHandleGet | Gets the connection handle of a HTTP connection. |
TCPIP_HTTP_NET_ConnectionIndexGet | Gets the index of the HTTP connection. |
TCPIP_HTTP_NET_DynamicFileInclude | Dynamically includes/writes a file to the HTTP connection. |
Data Types and Constants | |
_tag_TCPIP_HTTP_NET_USER_CALLBACK | HTTP user implemented callback data structure. |
TCPIP_HTTP_DYN_ARG_DCPT | HTTP dynamic argument descriptor. |
TCPIP_HTTP_DYN_ARG_TYPE | HTTP supported dynamic variables argument types. |
TCPIP_HTTP_DYN_PRINT_RES | Dynamic print result when a dynamic variable print callback function returns; |
TCPIP_HTTP_DYN_VAR_DCPT | HTTP dynamic variable descriptor. |
TCPIP_HTTP_DYN_VAR_FLAGS | HTTP supported dynamic variables flags. |
TCPIP_HTTP_NET_CONN_HANDLE | HTTP connection identifier, handle of a HTTP connection |
TCPIP_HTTP_NET_EVENT_TYPE | HTTP reported run-time events. |
TCPIP_HTTP_NET_IO_RESULT | Result states for execution callbacks |
TCPIP_HTTP_NET_MODULE_CONFIG | HTTP module dynamic configuration data |
TCPIP_HTTP_NET_MODULE_FLAGS | HTTP module configuration flags Multiple flags can be OR-ed |
TCPIP_HTTP_NET_READ_STATUS | Result states for TCPIP_HTTP_NET_ConnectionPostNameRead, TCPIP_HTTP_NET_ConnectionPostValueRead and TCPIP_HTTP_NET_ConnectionPostReadPair |
TCPIP_HTTP_NET_STATUS | Supported Commands and Server Response Codes |
TCPIP_HTTP_NET_USER_CALLBACK | HTTP user implemented callback data structure. |
TCPIP_HTTP_NET_USER_HANDLE | HTTP user handle. |
TCPIP_HTTP_NET_ConnectionPostReadPair | Reads a name and value pair from a URL encoded string in the network transport buffer. |
TCPIP_HTTP_SSI_ATTR_DCPT | HTTP SSI attribute descriptor. |
TCPIP_HTTP_SSI_NOTIFY_DCPT | HTTP SSI notification descriptor. |
_HTTP_NET_H | This is macro _HTTP_NET_H. |