llocate our ethernet header private Ip4 ip = new Ip4(); // Preallocat IP version 4 header public void nextPacket(PcapPacket packet, Object user) { if (packet.hasHeader(eth)) { System.out.printf("ethernet.type=%X\n", eth.type()); } if (packet.hasHeader(ip)) { System.out.println("ip.version=%d\n", ip.version()); } }
Accessing a subheader such as ip options
You can also access sub headers, usually supplied as options by the protocol during transmission.
private Ip4 ip = new Ip4(); // Preallocat IP version 4 header private Ip4.Timestamp timestamp = new Ip4.Timestamp(); // Optional header public void nextPacket(PcapPacket packet, Object user) { if (packet.hasHeader(ip) && ip.hasSubHeader(timestamp)) { System.out.println("ip.version=%d\n", ip.version); System.out.println("timestamp optional header length=%d\n", timstamp.length()); } A couple of points about the sub header example. Notice that we preallocated a Timestamp header, which is defined from within Ip4 class itself, but is a separate class on its own none the less. Next we first check if Ip4 header is present at all in the packet, peer it if exists (combined hasHeader and getHeader accessor method) and as a second step we check with the Ip4 header if it has an optional header using
ip.hasSubHeader(timestamp). If the method returns true, it also peers the sub header timestamp with the appropriate packet data buffer where the optional header resides.
Formatting packet for output
A packet can easily be formatted for textual output. Any supported formatter, such as TextFormatter or XmlFormatter can be used to format a packet for output. Also JPacket.toString() method uses an internal StringBuilder based TextFormatter that formats the packet for textual output in a string buffer. At this time both ip and timestamp header instances are properly intialized and can be used to access their respective headers.
JPacket packet = // From out handler TextFormatter out = new TextFormatter(System.out); out.format(packet); // Send pretty output to stdout // Or save time System.out.println(packet.toString()); // Use internal TextFormatter
Packet's lifecycle
A PcapPacket is made up of 3 parts:
- Packet data buffer - peered with packet object itself
- Packet state - peered with packet state object
- PcapCapture header - peered with packet header object
Each part of the packet is managed independently, that is either part can be initialized or not. Either part can point to any memory location, including a large single buffer of contigues bytes that contains all 3 parts, header, state and packet data. There are various methods supplied by PcapPacket that allow an external buffer to be peered with all 3 parts of the packet. There are also many methods for transfering (deep copy) the data to and from buffers.
All of these components are stored in native memory in native C structures that are peered with the packet API classes. The classes, managed by JMemory class are referencing native memory locations. Any native method that is called upon in the PcapPacket class or its base classes, will perform those operations on the peered structure and data.
When a packet is delivered from either Pcap.loop or Pcap.dispatch methods, the capture header, packet state and packet data all point to different unrelated memory locations. That is, capture header is peered with the libpcap supplied pcap_pkthdr structure, Packet data buffer (the packet itself) is peered with the data buffer supplied by pcap and the packet state is peered with its packet_state_t structure as supplied by the JScanner, typically out of its internal buffer. None of these default memory locations are persistent for very long time. Both libpcap and JScanner buffers are round robin buffers that eventually wrap around and reuse previously dispatched memory.
These temporary packets are only suitable for immediate use. That is if the packets are processed immediately when received and then discarded, they do not need to be preserved. If a packet is to be put on a queue and for later processing, the packet needs to preserve its state. That requires a physical copy of all 3 components of the packet to a new memory location. The most efficient way to store the new packet is to allocate a memory buffer large enough to hold all of the packets state and data out of a JMemoryPool. The JPacket provides a default singleton memory pool out of which all packets allocate memory out of for the required space.
Advanced topiccs
Below are several sections that describe the lifecycle of a packet in more depth. For simply usage, the termporary packets can be used immediately in the handler and then the packets can be discarded. For more advanced usage lets go into the detail of how packet data can be copied, preserved and peered to one another.
Perserving packet's state and data
In order to preserve packet's state and data a deep copy needs to be performed of all 3 components of he packet. PcapPacket class provides several
PcapPacket.transferTo methods that perform deep copies of the packet. For efficiency reasons, each transferTo method are designed to copy data into a memory buffer of larger size. The packet state and data are copied to the buffer with the following layout within the buffer:
+----------+-----+----+ |PcapHeader|State|Data| +----------+-----+----+
The buffer to which this copy takes place can be an external buffer or an internally allocated one by the packet class itself. As stated before, packet's use an interal singleton memory pool to allocate memory out of more efficiently. This memory allocates large native memory blocks which are then sub divided further and given out by the memory pool on a per request basis. All the copies are done natively by low level native copy routines, not in java space for maximum performace.
The easiest way to copy packet contents as received, for example, from PcapPacketHandler, is to pass the temporary packet to the PcapPacket constructor which will automatically allocate new space for the packet state and data and perform a deep copy. The new packet immediately becomes usable and is permanently stored in memory with its state and data, until garbage collected. Here is an example of a PcapPacketHandler that copies the temporary packet to new permanent one:
pulic void nextPacket(PcapPacket packet, Queue<PcapPacket> queue) { PcapPacket permanent = new PcapPacket(packet); queue.offer(permanent); }
Alternative is to reused another packet and transfer the temporary packets state and data to it or create a new unitiatialized packet with new PcapPacket(JMemory.Type.POINTER) constructor and subsequently perform PcapPacket.transferTo(PcapPacket) call to copy the contents. In the first case where an existing packet is being reused, if that packet already contains a large enough memory buffer to hold the state and data of the temporary packet, that buffer is reused. Otherwise a new buffer is allocated out of the default memory pool. Here is an exmaple: *
final PcapPacket permanent = new PcapPacket(Type.POINTER); pulic void nextPacket(PcapPacket packet, Queue<PcapPacket> queue) { permanent.transferStateAndData(packet); // Or packet.transferTo(permanent); } In either case, any existing buffer previously allocated in the permanent packet if its big enough to hold the state and data of the packet, is reused, saving time on memory allocation. You can also manually allocate a large buffer and reuse a packet:
final PcapPacket permanent = new PcapPacket(64 * 1024); // Preallocate 64K pulic void nextPacket(PcapPacket packet, Queue<PcapPacket> queue) { permanent.transferStateAndData(packet); // Or packet.transferTo(permanent); } In this example, the packet buffer will always be large enough and resused. But still this is a semi permanentn state.
Yet another alternative is to store the contents of the packet in an external buffer such as ByteBuffer, JBuffer or simply a byte[] and then at an appropriate time, transfer the data back or peer the external buffer with a packet object. Only the byte[] buffer type and ByteBuffer backed by a byte array, can not be peered directly with a packet as only buffer sources that are native memory based can be peered. All external buffer types can be copied back into a packet, if peering is not required. New memory space is allocated for the copy. Here is an example:
pulic void nextPacket(PcapPacket packet, Queue<PcapPacket> queue) { JBuffer jbuf = new JBuffer(packet.getTotalSize()); packet.transferTo(jbuf); // Or ByteBuffer bbuf = ByteBuffer.allocateDirect(packet.getTotalSize()); packet.transferTo(bbuf); // Or byte[] babuf = new byte[packet.getTotalSize())]; packet.transferTo(babuf); } In all 3 cases, complete the packet's state and data buffer are copied to external buffer.
Initializing packet from an external buffer
Packet state and data can be preseved in an external buffer large enough to hold the entire packet with its state. PcapPacket class provides transferStateAndData and peer methods that allow the external packet data to be either copied into a packet or the packet be peered directly with the external buffer. Peering does not need to allocate memory to hold the packet state, but its state and data are directly read out of the extern buffer. If you change the contents of the external buffer, the packet's state and data will change as well. Care must be take with a direct reference to an external buffer, as its easy to override sensitive data causing the packet to behave wildly and unexpectidly.
JMemory class prevents buffer overrun attacks and any access to memory that has not been allocated. a direct reference. Here is an example:
pulic void nextPacket(PcapPacket packet, Queue<PcapPacket> queue) { JBuffer jbuf = new JBuffer(packet.getTotalSize()); packet.transferTo(jbuf); // Or ByteBuffer bbuf = ByteBuffer.allocateDirect(packet.getTotalSize()); packet.transferTo(bbuf); // Or byte[] babuf = new byte[packet.getTotalSize())]; packet.transferTo(babuf); PcapPacket p1 = new PcapPacket(jbuf); // Deep copy PcapPacket p2 = new PcapPacket(Type.POINTER); // Uninitialized bbuf.flip(); // Have to flip the buffer to access the just written contents p2.peer(bbuf); // No copies, peered directly with external buffer PcapPacket p3 = new PcapPacket(Type.POINTER); // Uninitialized p3.transferStateAndData(babuf); // Deep copy - byte[] buffers can not be peered PcapPacket p4 = new PcapPacket(Type.POINTER); // Uninitialized p4.peer(p3); // both point at same internal memory space } The above example demonstrates 3 different ways that data from an external buffer can be either copied or peered with a new packet object. In all cases the data and state were transfered from the temporary packet received by the handler to a more permenant buffer and then packet. An interesting scenerio occures with packet p4. Lets take a closer look.
First, p3 is created unitialized, meaning that packet header, state and data are null pointers at this time, they don't point to anything and any accessor method used will immediately throw a NullPointerException. Second, the byte[] external buffer is copied into newly allocate memory space by p3. The packet is intiailized to pointer at its internal buffer for the header, state and packet data. Then we create p4, also unitialized and in the following step we peer p4 to p3. That is p4 points at the exact same memory location for packet's header, state and data. No new memory was allocated and changing the contents in either packet, p3 or p4, will have immediate effect on the other packet. Another words, both p3 and p4 are peered to the same internal memory space.
@author Mark Bednarczyk
@author Sly Technologies, Inc.
@see JMemoryPool