1.2.1.13 HTTP Net Module

The HTTP Net Web Server module allows a development board to act as a Web server. This facilitates an easy method to view status information and control applications using any standard Web browser. It uses the Networking Presentation Layer to integrate with an external encryption services provider (usually wolfSSL) allowing for secure connections.

Note: The HTTP_NET server supports encrypted communication by using the Networking Presentation layer to integrate an external encryption services provider (usually wolfSSL) for TLS support.

Abstraction Model

HTTP NET Abstraction Model

HTTP Net Features

The HTTP Net module is a HTTP server module that works using the Networking Presentation Layer of the MPLAB Harmony framework. This allows it to support both encrypted and plain text connections.

Some of the key features of the HTTP Net module include:

  • The updated module implements HTTP 1.1 persistent connections by default, using the “Transfer-Encoding: chunked” HTTP header. It can be configured to operate with closing the connections like the previous HTTP server.

  • The application has to dynamically register its HTTP processing functions with the HTTP Net server using the TCPIP_HTTP_NET_UserHandlerRegister call. The HTTP Net server no longer relies on well known function names.

  • The HTTP Net server supports either secure or plain text connections, but not both simultaneously. The specification of a secure or non-secure connection can be easily done by:

    • Selecting the server listening port: 80 or 443

    • Using the module configuration flags

  • The HTTP Net module reports run-time events that could be caught by the user application using the “eventReport” registered function that carries a TCPIP_HTTP_NET_EVENT_TYPE enumerated type

Dynamic Variables Processing

Parsing of the dynamic variables is done at run-time. The web pages could be changed at run-time, without the need for rebuilding the HTTP Net server. This allows the TCP/IP Stack to run as a library and only the application code to be changed accordingly. The application is required to register a “dynamicPrint” with the HTTP Net server. This function will be called at run-time to process the dynamic variables.

The supported syntax is var_name(param1, param2, …). When parsing a string like this within a web page, the HTTP Net server will invoke the “dynamicPrint” function using a TCPIP_HTTP_DYN_VAR_DCPT data structure that specifies the dynamic variable name, its number and type of parameters as well as each parameter value.

For example, the variable myVariable(2,6) will generate the “dynamicPrint” call with the following parameters:

varDcpt.dynName = "myVariable"; 
varDcpt.nArgs = 2; 
varDcpt.dynArgs->argType = TCPIP_HTTP_DYN_ARG_TYPE_INT32; 
varDcpt.dynArgs->argInt32 = 2; 
(varDcpt.dynArgs + 1)->argType = TCPIP_HTTP_DYN_ARG_TYPE_INT32; 
(varDcpt.dynArgs + 1)->argInt32 = 6; 

String and int32_t variable types are currently supported.

The application needs to return a TCPIP_HTTP_DYN_PRINT_RES result specifying if it’s done with processing or it needs to be called again.

Applications no longer have direct access to the underlying transport socket for a HTTP connection. All of the data write operations need to go through the required API function: TCPIP_HTTP_NET_DynamicWrite. This function actually just attaches the data to be written to the connection, using data descriptors. The actual output occurs only when all data is gathered and the data size is known.

Buffers used in a TCPIP_HTTP_NET_DynamicWrite need to be persistent until the moment when the output is generated. RAM buffers are supported by using a “dynamicAck” (TCPIP_HTTP_NET_DynAcknowledge) type function. Once the output operation is completed the HTTP Net module will call back into the application indicating that the corresponding buffer is no longer used and can be freed/reused.

In situations where the dynamic variable print function needs to perform additional write operations, or simply needs to be called again, it must a special result code: TCPIP_HTTP_DYN_PRINT_RES_PROCESS_AGAIN/TCPIP_HTTP_DYN_PRINT_RES_AGAIN. Typically this is used when outputting large amounts of data that cannot fit into one single buffer write operation or when the data is not available all at once.

Note: The parsing of the files for dynamic content is done at run time. Therefore when generating the image, files that contain dynamic content should not be compressed!

Include Files Processing

The HTTP Net module dynamically parses the web pages and the file inclusion is done by the module itself, completely transparent for the application.

The included files can contain dynamic variables that are parsed at run time.

File Processing

File processing consists of the following:

  • An included file can include other files that have dynamic variables

  • Application can include a file as part of its dynamic variable processing using the TCPIP_HTTP_NET_ConnectionFileInclude()API. That file can include other files.

  • The only limit to the depth of inclusion is the memory resources and the TCPIP_HTTP_NET_MODULE_CONFIG:: maxRecurseLevel setting. A TCPIP_HTTP_NET_EVENT_DEPTH_ERROR event will be generated if the maximum depth is exceeded at run-time.

SSI Processing

At run-time, the HTTP Net server processes a subset of the Server Side Includes (SSI) commands. SSI is a portable way to add small amounts of dynamic content on web pages in a standard way that is supported by most HTTP servers. Currently the #include, #set and #echo commands are supported. Other commands will be eventually added. This allows a better compatibility with other existing HTTP servers and is targeted at the removal of proprietary extensions.

Whenever executing an SSI command within a web page, the HTTP Net server will notify the application by using the registered SSI callback: ssiNotify. This callback contains a TCPIP_HTTP_SSI_NOTIFY_DCPT pointer to a structure describing the SSI context needed for processing:

  • File name the SSI command belongs to

  • The SSI command line

  • Number of SSI attributes and their values, etc.

The application can do any modifications it chooses to and returns a value instructing the HTTP Net server if the processing of the command is necessary or not.

SSI include Command

The SSI Include command allows the dynamic inclusion of a file. The supported syntax is:

<!--#include virtual="file_name" --> or <!--#include file="file_name" -->. 

Currently the arguments for both “virtual” and “file” commands are passed unaltered to the SYS_FS, so they behave identically. However it is recommended that the original SSI significance for these keywords should be maintained:

  • Use “virtual” for specifying a URL relative to the document being server

  • Use “file” for a file path, relative to the current directory (it cannot be an absolute path) The inc:file_name keyword is maintained for backward compatibility. However the SSI include command should be preferred.

SSI set Command

The SSI set command allows to dynamically set a SSI variable value. The supported syntax is:

<!--#set var="v_name" value="v_value" -->. 

String or integer variables are supported. Variable reference is also supported:

<!--#set var="n_name" value="$otherVar" -->. 

This command will create or update the variable n_name to have the value of the variable otherVar, if it exists.

A new SSI variable will be created if a variable having the name v_name does not exist. If the variable v_name already exists, it will have its value updated as the result of this command.

An existing variable can be deleted using the empty value set command:

<!--#set var="v_name" value="" -->.

SSI echo Command The SSI echo command allows the dynamic print of a SSI variable. The supported syntax is:

<!--#echo var="v_name" -->.

If the application ssiNotify exists, the HTTP Net server will call it and the application may choose to change the current value dynamically. If ssiNotify returns false, the HTTP Net server will display the current value of the variable v_name as part of the current page.

The SSI API can be used to evaluate or change the current value of the SSI variables:

  • TCPIP_HTTP_NET_SSIVariableGet

  • TCPIP_HTTP_NET_SSIVariableSet

  • TCPIP_HTTP_NET_SSIVariableDelete

The maximum number of SSI variables is under the control of the application by using the configuration parameter: TCPIP_HTTP_NET_SSI_VARIABLES_NUMBER.

mpfs2.jar Utility

Dynamic Variable Processing

The HTTP Net server does not use the DynRecord.bin and FileRcrd.bin files for processing the dynamic variables. The parsing and processing of dynamic variables is done at run-time. Therefore, the mpfs2.jar utility is no longer necessary, other than to generate a mpfs_img2.c image file for the NVM MPFS image.

Application Custom Support Files

The application custom support files generated with MHC have new names: custom_http_net_app.c and http_net_print.c. However, the http_net_print.c file is not generated by the mpfs2 utility and is maintained only for easy comparison between HTTP and HTTP Net application processing. The custom_http_net_app.c file is entirely generated using a MHC template file and it is not dynamically updated in any way by the mpfs2 utility.

Generated File Name

Currently the name of the generated file for MPFS image is maintained unchanged: mpfs_img2.c.

MPFS Image Generation for Internal NVM Storage

The mpfs2 utility can still be used to generate the MPFS image for internal NVM storage. It can also be useful because it parses the web pages and comes out with the http_print.c file that contains the list of TCPIP_HTTP_Print functions. This can be helpful in gathering info about the dynamic variables that are contained within the web pages.

Note: For demonstration applications that use SSI, the file inclusion is now done in a standard way using HTML (i.e., .htm) files. Therefore, when generating the image, *.htm must be added to Advanced Settings > Do Not Compress.

Library Interface

NameDescription
Macros
TCPIP_HTTP_NET_CACHE_LENMax lifetime (sec) of static responses as string
TCPIP_HTTP_NET_CONFIG_FLAGSDefine the HTTP module configuration flags Use 0 for default See HTTP_MODULE_FLAGS definition for possible values
TCPIP_HTTP_NET_COOKIE_BUFFER_SIZESize of the buffer used for sending the cookies to the client. Should be able to accommodate the longest cookie response. Otherwise the cookies will be truncated.
TCPIP_HTTP_NET_DEFAULT_FILEIndicate what HTTP file to serve when no specific one is requested
TCPIP_HTTP_NET_DYNVAR_ARG_MAX_NUMBERmaximum number of arguments for a dynamic variable
TCPIP_HTTP_NET_DYNVAR_DESCRIPTORS_NUMBERhow many buffers descriptors for dynamic variable processing they are independent of the HTTP connection number all the HTTP connections use from the dynamic descriptors pool
TCPIP_HTTP_NET_DYNVAR_MAX_LENmaximum size for a complete dynamic variable: name + args must be <= TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_SIZE! If it is much larger than needed then inefficiency occurs when reading data from the file and then discarding it because a much larger than needed data buffer was read
TCPIP_HTTP_NET_FILE_UPLOAD_ENABLEConfigure MPFS over HTTP updating Comment this line to disable updating via HTTP
TCPIP_HTTP_NET_FILE_UPLOAD_NAMEThis is macro TCPIP_HTTP_NET_FILE_UPLOAD_NAME.
TCPIP_HTTP_NET_FIND_PEEK_BUFF_SIZESize of the peek buffer to perform searches into. If the underlying transport layer supports offset peak operation with a offset, the value could be smaller (80 characters, for example); otherwise, a one time peek is required and the buffer should be larger - recommended to be close to the size of the socket RX buffer.
TCPIP_HTTP_NET_MAX_CONNECTIONSMaximum numbers of simultaneous supported HTTP connections.
TCPIP_HTTP_NET_MAX_DATA_LENDefine the maximum data length for reading cookie and GET/POST arguments (bytes)
TCPIP_HTTP_NET_MAX_HEADER_LENThe length of longest header string that can be parsed
TCPIP_HTTP_NET_MAX_RECURSE_LEVELThe maximum depth of recursive calls for serving a web page: no dynvars files: 1; file including a file: 2 ; if the include file includes another file: +1 ; if a dyn variable: +1; Default value is 3;
TCPIP_HTTP_NET_RESPONSE_BUFFER_SIZESize of the buffer used for sending the response messages to the client. Should be able to accommodate the longest server response: Default setting should be 300 bytes
TCPIP_HTTP_NET_SKT_RX_BUFF_SIZEDefine the size of the RX buffer for the HTTP socket Use 0 for default TCP socket value The default recommended value for high throughput is > 2MSS (3 KB). The performance of a socket is highly dependent on the size of its buffers so it's a good idea to use as large as possible buffers for the sockets that need high throughput.
TCPIP_HTTP_NET_SKT_TX_BUFF_SIZEDefine the size of the TX buffer for the HTTP socket Use 0 for default TCP socket value The default recommended value for high throughput is > 2MSS (3 KB). The performance of a socket is highly dependent on the size of its buffers so it's a good idea to use as large as possible buffers for the sockets that need high throughput.
TCPIP_HTTP_NET_TASK_RATEThe HTTP task rate, ms The default value is 33 milliseconds. The lower the rate (higher the frequency) the higher the module priority and higher module performance can be obtained The value cannot be lower than the TCPIP_STACK_TICK_RATE.
TCPIP_HTTP_NET_TIMEOUTMax time (sec) to await for more HTTP client data in an active connection state before timing out and disconnecting the socket
TCPIP_HTTP_NET_USE_AUTHENTICATIONEnable basic authentication support
TCPIP_HTTP_NET_USE_COOKIESEnable cookie support
TCPIP_HTTP_NET_USE_POSTDefine which HTTP modules to use If not using a specific module, comment it to save resources Enable POST support
TCPIP_HTTP_NET_CHUNK_RETRIESretry limit for allocating a chunk from the pool If more retries are not successful the operation will be aborted
TCPIP_HTTP_NET_CHUNKS_NUMBERnumber of chunks that are created It depends on the TCPIP_HTTP_NET_MAX_RECURSE_LEVEL and on the number of connections Maximum number should be TCPIP_HTTP_NET_MAX_CONNECTIONS * TCPIP_HTTP_NET_MAX_RECURSE_LEVEL i.e. TCPIP_HTTP_NET_MODULE_CONFIG::nConnections * TCPIP_HTTP_NET_MODULE_CONFIG::nChunks All the chunks are in a pool and are used by all connections
TCPIP_HTTP_NET_DYNVAR_PROCESSThis symbol enables the processing of dynamic variables Make it evaluate to false (0) if dynamic variables are not needed All the following symbols referring to dynamic variables are relevant only when TCPIP_HTTP_NET_DYNVAR_PROCESS != 0
TCPIP_HTTP_NET_DYNVAR_PROCESS_RETRIESretry limit for a dynamic variable processing ths puts a limit on the number of times a dynamic variable "dynamicPrint" function can return TCPIP_HTTP_DYN_PRINT_RES_AGAIN/TCPIP_HTTP_DYN_PRINT_RES_PROCESS_AGAIN and avoids having the HTTP code locked up forever. If more retries are attempted the processing will be considered done and dynamicPrint function will not be called again
TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_RETRIESRetry limit for allocating a file buffer from the pool. If more retries are not successful the operation will be aborted.
TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_SIZESize of the buffer used for processing HTML, dynamic variable and binary files. For dynamic variable files it should be able to accommodate the longest HTML line size, including CRLF!
TCPIP_HTTP_NET_FILE_PROCESS_BUFFERS_NUMBERNumber of file buffers to be created; These buffers are used to store data while file processing is done They are organized in a pool Each file being processed needs a file buffer and tries to get it from the pool If a buffer is not available, the HTTP conenction will wait for one to become available. Once the file is done the file buffer is released and could be used by a different file The number depends on the number of files that are processed in parallel To avoid deadlock the number should be >= than the number of... more
TCPIP_HTTP_NET_FILENAME_MAX_LENmaximum size of a HTTP file name with the path removed from the file name one extra char added for the string terminator
TCPIP_HTTP_NET_SSI_ATTRIBUTES_MAX_NUMBERmaximum number of attributes for a SSI command most SSI commands take just one attribute/value pair per line but multiple attribute/value pairs on the same line are allowed where it makes sense
TCPIP_HTTP_NET_SSI_CMD_MAX_LENmaximum size for a SSI command line: command + attribute/value pairs must be <= TCPIP_HTTP_NET_FILE_PROCESS_BUFFER_SIZE! If it is much larger than needed then inefficiency occurs when reading data from the file and then discarding it because a much larger than needed data buffer was read
TCPIP_HTTP_NET_SSI_ECHO_NOT_FOUND_MESSAGEmessage to echo when echoing a not found variable
TCPIP_HTTP_NET_SSI_PROCESSThis symbol enables the processing of SSI commands Make it evaluate to false (0) if SSI commands are not needed All the following symbols referring to SSI commands are relevant only when TCPIP_HTTP_NET_SSI_PROCESS != 0
TCPIP_HTTP_NET_SSI_STATIC_ATTTRIB_NUMBERnumber of static attributes associated to a SSI command if the command has more attributes than this number the excess will be allocated dynamically
TCPIP_HTTP_NET_SSI_VARIABLE_NAME_MAX_LENGTHmaximum length of a SSI variable name any excess characters will be truncated Note that this can result in multiple variables being represented as one SSI variable
TCPIP_HTTP_NET_SSI_VARIABLE_STRING_MAX_LENGTHmaximum size of a SSI string variable value any excess characters will be truncated Note that the variable value requires SSI storage that's allocated dynamically Also, this value determines the size of an automatic (stack) buffer when the variable is echoed. If this value is large, make sure you have enough stack space.
TCPIP_HTTP_NET_SSI_VARIABLES_NUMBERmaximum number of SSI variables that can be created at run time These variables are stored in an internal hash. For max. efficiency this number should be a prime.
TCPIP_HTTP_NET_CONNECTION_TIMEOUTPersistent connection idle timeout, in seconds If a persistent connection is idle for more that this timeout the server will close it. Usually the client closes the connection. Using this timeout value the server can close the its side of the connection to avoid all connections to remain indefinitely open if the client misbehaves. Use 0 to never timeout. The timeout value has to be <= 32767 seconds.
HTTP_NET_CONFIG_HThis is macro HTTP_NET_CONFIG_H.
TCPIP_HTTP_NET_FREE_FUNCHTTP corresponding deallocation function, free style This is the function the HTTP will call for freeing the allocated memory Use standard C library 'free' or 0 as a default If it's 0, HTTP will use the allocation functions passed in at the stack initialization
TCPIP_HTTP_NET_MALLOC_FUNCHTTP allocation function, malloc style This is the function the HTTP will call to allocate memory needed for: dynamic variables, SSI or file uploads. Use standard C library 'malloc' or 0 as a default If it's 0, HTTP will use the allocation functions passed in at the stack initialization
Functions
TCPIP_HTTP_NET_ConnectionByteCountDecDecrements the connection byte count.
TCPIP_HTTP_NET_ConnectionByteCountSetSets how many bytes are available to be read.
TCPIP_HTTP_NET_ConnectionCallbackPosSetSets the callback position indicator.
TCPIP_HTTP_NET_ConnectionDiscardDiscards any pending data in the connection RX buffer.
TCPIP_HTTP_NET_ConnectionFlushImmediately transmits all connection pending TX data.
TCPIP_HTTP_NET_ConnectionHasArgsSetSets whether there are get or cookie arguments.
TCPIP_HTTP_NET_ConnectionIsAuthorizedSetSets the authorized state for the current connection.
TCPIP_HTTP_NET_ConnectionNetHandleReturns the network handle of the current connection.
TCPIP_HTTP_NET_ConnectionPeekReads a specified number of data bytes from the connection RX buffer without removing them from the buffer.
TCPIP_HTTP_NET_ConnectionPostSmSetSet the POST state machine state.
TCPIP_HTTP_NET_ConnectionReadReads an array of data bytes from a connection's RX buffer.
TCPIP_HTTP_NET_ConnectionReadBufferSizeReturns the size of the connection RX buffer.
TCPIP_HTTP_NET_ConnectionReadIsReadyDetermines how many bytes can be read from the connection RX buffer.
TCPIP_HTTP_NET_ConnectionStatusSetSets HTTP status.
TCPIP_HTTP_NET_ConnectionStringFindHelper to find a string of characters in an incoming connection buffer.
TCPIP_HTTP_NET_ConnectionUserDataSetSets the user data parameter for the current connection.
TCPIP_HTTP_NET_DynamicWriteWrites a data buffer to the current connection
TCPIP_HTTP_NET_DynamicWriteStringHelper for writing a string within a dynamic variable context.
TCPIP_HTTP_NET_TaskStandard TCP/IP stack module task function.
TCPIP_HTTP_NET_URLDecodeParses a string from URL encoding to plain text.
TCPIP_HTTP_NET_UserHandlerDeregisterDeregisters a previously registered HTTP user handler.
TCPIP_HTTP_NET_UserHandlerRegisterRegisters a user callback structure with the HTTP server
TCPIP_HTTP_NET_ActiveConnectionCountGetGets the number of active (and inactive) connections.
TCPIP_HTTP_NET_ArgGetLocates a form field value in a given data array.
TCPIP_HTTP_NET_ConnectionByteCountGetReturns how many bytes are available to be read.
TCPIP_HTTP_NET_ConnectionCallbackPosGetReturns the callback position indicator.
TCPIP_HTTP_NET_ConnectionDataBufferGetReturns pointer to connection general purpose data buffer.
TCPIP_HTTP_NET_ConnectionFileGetGet handle to current connection's file.
TCPIP_HTTP_NET_ConnectionHasArgsGetChecks whether there are get or cookie arguments.
TCPIP_HTTP_NET_ConnectionIsAuthorizedGetGets the authorized state for the current connection.
TCPIP_HTTP_NET_ConnectionPostSmGetGet the POST state machine state.
TCPIP_HTTP_NET_ConnectionSocketGetGet the socket for the current connection.
TCPIP_HTTP_NET_ConnectionStatusGetGets HTTP status.
TCPIP_HTTP_NET_ConnectionUserDataGetGets the user data parameter for the current connection.
TCPIP_HTTP_NET_ConnectionPostNameReadReads a name from a URL encoded string in the network transport buffer.
TCPIP_HTTP_NET_ConnectionPostValueReadReads a value from a URL encoded string in the network transport buffer.
TCPIP_HTTP_NET_ConnectionDataBufferSizeGetReturns the size of the connection general purpose data buffer.
TCPIP_HTTP_NET_ConnectionDynamicDescriptorsReturns the number of dynamic variable descriptors
TCPIP_HTTP_NET_SSIVariableDeleteFunction to delete an SSI variable.
TCPIP_HTTP_NET_SSIVariableGetFunction to get access to an existing SSI variable.
TCPIP_HTTP_NET_SSIVariableGetByIndexFunction to get access to an existing SSI variable.
TCPIP_HTTP_NET_SSIVariableSetFunction to set an SSI variable.
TCPIP_HTTP_NET_SSIVariablesNumberGetFunction to get the number of the current SSI variables.
TCPIP_HTTP_NET_ConnectionHandleGetGets the connection handle of a HTTP connection.
TCPIP_HTTP_NET_ConnectionIndexGetGets the index of the HTTP connection.
TCPIP_HTTP_NET_DynamicFileIncludeDynamically includes/writes a file to the HTTP connection.
Data Types and Constants
_tag_TCPIP_HTTP_NET_USER_CALLBACKHTTP user implemented callback data structure.
TCPIP_HTTP_DYN_ARG_DCPTHTTP dynamic argument descriptor.
TCPIP_HTTP_DYN_ARG_TYPEHTTP supported dynamic variables argument types.
TCPIP_HTTP_DYN_PRINT_RESDynamic print result when a dynamic variable print callback function returns;
TCPIP_HTTP_DYN_VAR_DCPTHTTP dynamic variable descriptor.
TCPIP_HTTP_DYN_VAR_FLAGSHTTP supported dynamic variables flags.
TCPIP_HTTP_NET_CONN_HANDLEHTTP connection identifier, handle of a HTTP connection
TCPIP_HTTP_NET_EVENT_TYPEHTTP reported run-time events.
TCPIP_HTTP_NET_IO_RESULTResult states for execution callbacks
TCPIP_HTTP_NET_MODULE_CONFIGHTTP module dynamic configuration data
TCPIP_HTTP_NET_MODULE_FLAGSHTTP module configuration flags Multiple flags can be OR-ed
TCPIP_HTTP_NET_READ_STATUSResult states for TCPIP_HTTP_NET_ConnectionPostNameRead, TCPIP_HTTP_NET_ConnectionPostValueRead and TCPIP_HTTP_NET_ConnectionPostReadPair
TCPIP_HTTP_NET_STATUSSupported Commands and Server Response Codes
TCPIP_HTTP_NET_USER_CALLBACKHTTP user implemented callback data structure.
TCPIP_HTTP_NET_USER_HANDLEHTTP user handle.
TCPIP_HTTP_NET_ConnectionPostReadPairReads a name and value pair from a URL encoded string in the network transport buffer.
TCPIP_HTTP_SSI_ATTR_DCPTHTTP SSI attribute descriptor.
TCPIP_HTTP_SSI_NOTIFY_DCPTHTTP SSI notification descriptor.
_HTTP_NET_HThis is macro _HTTP_NET_H.